Italy faces a general election in early 2018 and a deluge of far-right groups are fighting for a place in a likely coalition Government.

In April 2016, the European Investigative Collaborations network of 12 major national news organisations received a dataset detailing shady business in football, and set about investigating the mass of rich contents.

This consisted of email inboxes, a bunch of PDF and Word documents, zip and rar archives, whatsapp conversations and encrypted hushmail communication. Out of 1.9TB of data, our team analyzed one terabyte of data, which yielded more than 6 million individual documents

It was the largest leak in the history of sport and was made available to the investigative team of German news magazine Der Spiegel.

One of the options for EIC journalists to search through the data was a server that we set up at the Romanian Centre of Investigative Journalism (RCIJ), using whatever hardware was at hand.

Specifically, a 2011 MacBook Pro with an after-market SSD, to which we hooked up a large USB drive. It turns out that the MacBook, with quad I7 processors, is a powerful beast, and could churn through the documents at an impressive rate.

We processed the data in stages, using a tool we built for this project called ‘Snoop’. To get started, we listed all the files in the dataset, and stored them in a PostgreSQL database. Then we extracted the text from the easy emails (from Apple Mail inboxes) and indexed them into elasticsearch, a versatile open-source search database.

Now we wanted to give our journalists access to the search engine so they could start their work.

A Virtual Private Network (VPN) is difficult to set up and support, and replicating the search engine on-premises for each of the 12 news organizations would require hardware and people to take care of each server. We didn't have the resources for that. So we set up a secure HTTPS website, two-factor authentication using the TOTP protocol, time-limited login sessions, rate limiting and traffic logging, which gave us enough confidence to open up the server to our network of journalists, over the public Internet.

We implemented a straightforward process for sign-up and account recovery: the user would get a URL, which contained a short-lived code, that opened a form page. Here they would choose a password, and set up a second factor for authentication, with a compatible phone app like Google Authenticator or Duo Mobile. This process went smoothly and saved us a lot of support work.

Then we went back to the dataset: we ran Tika, a tool that extracts text and metadata from a bunch of different file formats (PDFs, Word documents, etc). We also introduced the concept of file containers, so we could analyze email attachments and archive contents.

Emails were a particular challenge, because they have a tree-like structure of text parts and file attachments, with a bunch of headers with specific meanings, and they come in several file formats to boot. The email parser from Python’s standard library would handle anything that looked like standard RFC-822, and Apple Mail's "emlx" format is a simple adaptation of that. We converted Outlook files using msgconvert and readpst.

There were many scanned documents in the dataset, so early on, Der Spiegel made a big effort to process all PDFs and images through OCR - a process that retrieves the original text from printed documents. We added support in Snoop, matching the files by their unique MD5 checksum, and indexed the OCR output along with the original file.

Each feature added to Snoop would yield more documents for the search engine. To make sure we didn't miss anything, we re-processed the whole dataset every time, using caches to avoid duplicating expensive steps, like parsing emails, running Tika, and unpacking archives. Internally, each document has a unique identifier, so there are no duplicates in the index, and documents have stable URLs in the search engine.

We started out with a very basic user interface: a search box, highlighting for matches in search results, and clicking on a result opened a new tab with nothing more than a text-only preview of the document. As the project went on, we evolved the user interface (UI) with feedback from journalists, to a two-column layout with search results on the left and document preview on the right. Now we display a bunch of metadata for each document, along with links to download the original file, OCRed version (if available), and links to parent and child documents (e.g. email attachments, zip archive contents, and navigation of the original dataset's folder structure).

Journalists were incredibly forgiving with glitches and initial usability problems, as long as they could actually search through the files, and, over time, get access to more of the dataset, as we improved the processing toolchain. The whole group stayed in touch via Rocket.chat, an open-source clone of Slack, that we ran on a self-hosted Sandstorm server, and for the tech team, the questions, feedback and encouragement were invaluable. We hardly needed to set up monitoring in case the server went down - very soon we would receive worried (but friendly) messages and emails…

On the development team, coders Gabriel Vîjială, Dragoș Catarahia, Victor Avasiloaei and myself worked on the data processing and user interface, Coder Dan Achim integrated Hypothesis annotations, and Raluca Ciubotaru designed the user interface and helped us understand how journalists use the tool.

All of this work - document processing, indexing, the search interface, two-factor authentication and the signup process - is open source under the "hoover" umbrella project. We were in a unique position to build the tool, with constant user feedback, a large and varied real-life dataset, and hard publishing deadlines. Now we're smoothing out the rough edges so hoover can be used in other similar projects, both at EIC and in other places.

The source code is on GitHub - https://github.com/hoover - and includes an installation utility to get started quickly. If you try it out, we'd love to hear your experiences.

A cross-border investigation, by Cristian Niculescu, Eric Breitinger, Aleksandra Jolkina, Harry Karanikas, David Leloup, Dimitra Triantafillou and Stanimir Vaglenov, focusing on six European countries: Romania, Belgium, Bulgaria, Greece, Latvia and Switzerland.

This report finds that in Romania, despite having a National Health service that provides affordable medical care for all registered citizens, the Government is not paying for all necessary cancer drugs. 

Many patients must borrow money or sell their properties to pay for their own treatment, even if they have contributed to the public system all their lives through a national insurance scheme.

But while the state lacks funds, corruption and controversial practices that manipulate free trade rules are rife across the pharma sector.

The small prices on some medicines in Romania set by the state means pharma products destined for the Romanian market end up in retailers in Germany, Denmark, Poland or any other EU country where they can sell for higher prices.

For example, Mabthera is an injectable onocology drug that treats lymphatic cancer, manufactured by Swiss pharma giant Roche. One ampoule in Romania costs around 1,330 Euro (6,000 RON).

Roche sells 500 of these to the leading Romanian pharma distributors - Mediplus, AD Pharma, Polisano, Farmexim and Farmexpert, according to industry sources. From this figure, 495 boxes go immediately to Germany - securing a minimum of a 15 per cent profit on the input price. A small fraction of this number remain in Romania for patients.

The big pharma players re-export a great number of the commodities received from the producers, by ignoring national regulations.

No one can prove this intra-community trade is taking place because the distributors “forget” to report the movements of products to the Romanian authorities, as required by law since 2013.

The lack of reporting opens the door to medication fraud, such as the secondary sale of products across borders. The huge profits to be made from pharma allow big distribution companies to corrupt doctors and eliminate independent competition.

The scheme works like this:

A pharma distributor offers cash to an oncologist. The oncologist writes out prescriptions for a patient to be bought from a branch of a pharmacy that belongs to the same distributor.

There is no patient present as the pharmacist enters the details of the prescriptions into the computer system.

Then the pharmacy invoices the state health insurance authority CNAS to reimburse them for the cost.

Meanwhile the drug itself stays on the shelf.

This means the distributor can export the product to another company or associate in its group.

Therefore the distributor can double its money - once in the reimbursement from the Romanian state - and again on the sale of the product for export.

The visible and dramatic effect is that the vital medicines become unavailable to patients, so many die. 

The investigation details a 67 year-old woman from Romania who saves her husband from prostate cancer. Cristina Achimoiu regularly travels 200 km by train to Bucharest to sell her handmade wool garments, and uses the money for her husband's treatment.

Cristina has a pension of 400 lei (around 85 Euro) per month, ten times less than the amount of medicines needed by the patient. A journalist posted on Facebook the story of Cristina that later received helped from a lot of Romanians, impressed by her sacrifice. But this is the exception.

Mioara Dumitrescu was 47 years old when she was diagnosed with brain cancer, in 2011. She published a letter attacking the health authorities who refused her Avastin treatment and she asked for the state to consider her a candidate for euthanasia.

The drug is not included on the list of free medicines the state can provide in Romania. Drugs like Avastin, Herceptin, Interferon, Zoladex or Tarceva are issued only after special or complicated approval protocols.

In 2012 Mioara Dumitrescu sued the state and won the trial with the state health insurance authority CNAS. After lengthy procedures, CNAS finally approved the 4,000 Euro per month needed for her treatment. This was too late, as Mioara died in 2013.

This is a cross-border investigation developed with support from JournalismFund.eu 

Cluj-born mother and cancer survivor Eleonora suffered decades of physical abuse from her husband.

After visiting doctors and gaining evidence of abuse she eventually managed to get a divorce.

However her former husband refused to leave the house owned by Eleonora's parents.

This meant they stayed living together - and he insisted on sleeping in the same bed as her.

In the same way as before the divorce, he would force her to have sex with him.

His argument was that God does not accept a divorce.

Therefore she was still his wife in the eyes of God - so had to submit to his will.

“If I didn’t want to have sex,” says Eleonora, “he would make a scandal and wake up the whole house - my parents, my children, no matter the hour, so I had to give in.”

On a woman’s psychological and physical well being, rape can produce one of the greatest traumas - and victims rarely report the crime.

But when rape happens inside a marriage, acknowledging the abuse becomes harder for the victim.

In Romania there are no statistics on the number of wives raped by their husbands – leading experts to question whether any man has ever been convicted for marital rape.

This is a form of domestic violence, but experts argue that few women know that sex without consent with their husband is punishable by law.

One reason women do not report marital rape, as opposed to other forms of physical violence, is that the crime involves a deeper level of emotional and sexual abuse.

There is also the belief that through marriage, Romanian women make a vow of submission to their husbands in every respect.

This differs depending on the couples’ education or social background, but the country’s patriarchal culture transmits to a staggering majority of women the message that they must comply with the man.

When women ask for help to the police, to social workers or NGOs, because they have been physically abused, they have to fill in a form detailing the types of abuse they have experienced. Rarely is marital rape mentioned by the women at this stage.

Experts argue that this happens because rape is considered either a taboo subject or is seen as such a common occurrence in the marriage that women do not believe it constitutes a form of abuse.

During counseling with a therapist, it takes a few sessions and a bond to form between the victim and therapist, before the victim can even implicitly speak about marital rape.

It is also hard to find evidence for marital rape.

Many women consider the forced sexual act is a shame and only see a doctor or a specialist after they have washed and cleaned themselves. 

Women also submit in order not to suffer injuries. In the absence of lesions that might indicate violence or residue that could determine ejaculation, proving such a crime is near-impossible.

Due to the communist regime, Romania skipped over the period in west Europe, when activists and feminist organizations brought issues of abuse to the public attention.

Instead a top-down system worked, where the state emancipated women in the workplace, but at home women retained the same roles, often in submission to the husband.

Nevertheless at work, it was not a egalitarian utopia. There were many instances of sexual harassment which the bosses covered up. 

Since 1989, many Romanian women stated they did not need feminism, because they could succeed alone or because it was seen as something that took women out of their natural space - the family - and robbed them of their femininity.

Now, to deal with women’s issues all of Romania needs to understand and admit what they are - women’s issues.

Other members of the family suffer the effects of violence, but it is a woman who is abused every 30 seconds in Romania.

To allow women to speak up about the abuses, the country should stop whispering about both domestic abuse and sex, so that women feel empowered to speak up about victimization.

Unless people open up about these issues, domestic violence stays the dirty family secret and marital rape the secret within that secret.

Romania’s tough new draft law endangering the privacy of communicating via telephone and the Internet has been labelled “tyranny” by Richard Stallman, American founder of the Free Software Foundation.

The EU country is proposing a law requiring users to register their identification numbers for Pre-Pay mobile phone cards.

Providers of free public WiFi hotspots will also have to “identify” their users, under the law.

A fierce defender of online privacy, Stallman called the new law “tyranny” in answer to a question at the Bucharest Coliberator conference in Bucharest on 7 June.

He added that people should defy the law by running their own WiFi networks that don’t require registration.

The Government did not submit the draft law to civil society nor the industry, in a move sure to incense privacy advocates.

The law would also force Romania’s 12 million users of pre-pay phone to register their number with an ID in the next six months or face disconnection.

At present, the details of the legal change are fuzzy.

It states any provider of a WiFi network in, say, a cafe, hotel, park or university, will have to “identify” its users and store the information for up to six months.

This could mean that a user of a WiFi hotspot while drinking a coffee or sitting on a bench will have to register with an ID or Passport.

This is accompanied by a new draft law on Cybersecurity, will make communications data on phone and Internet users available to nine Government agencies, including the secret services.

The push to introduce both laws shows a tightening of privacy on communication at the same time as a widening of surveillance capabilities by state bodies.  

“These laws show the true face of our Government’s desire for digital Romania,” says Bogdan Manolea, legal communications expert and executive director of the Association for Technology and the Internet. “A virtual space where a user of any communication device has to supply their personal data and where this data will be accessible directly, without a warrant, by all secret and non-secret services.” 

The justification for the law on pre-pay cards is that it eliminates a channel which terrorists or criminals could use to communicate. It makes surveillance of such targets easier.

However such criminals would be wise to the law and would use false or stolen cards or an intermediary.

They could also, writes Manolea, use encryption services which prevent anyone accessing the call.

This hurdle could be “even worse” for prosecutors and the intelligence community, he says.

The new law will also be a headache for any providers of public WiFi.

For example, Starbucks, the Hilton Hotel and McDonald’s may be obliged to store private data regarding their customers’ communication history.

Therefore the Golden Arches will be acting as a ‘data center’ for the Romanian secret service and its allies.

Is changing Christmas from 7 January to 25 December a mark of the westernisation of Moldova - or a Catholic plot against the Orthodox Church? Anthropologist Olga Cojocaru shares her research into the two Christmases of Moldova

Those familiar with the conflict around a proposed multi-billion-dollar gold mining project in Romania’s Apuseni mountains, Rosia Montana, are aware that the protestors demand to list Rosia Montana as an UNESCO Heritage site.

They believe if the historical mining area receives protected status, it will stop Canadian firm Rosia Montana Gold Corporation from constructing a mine on the site, which will see the destruction of four mountains and the use of cyanide in the extraction process. 

The project itself is currently on hold, so the protestors are pushing to use this window of opportunity to push for a UNESCO listing.

However Razvan Theodorescu, an ex-Minister of Culture and a senior member of scientific think tank the Romanian Academy, suggested before a parliamentary commission that such an idea was “laughable”.

But what exactly does UNESCO Heritage listing mean, and how comical is submitting Rosia Montana for such a listing?

The Convention Concerning the Protection of the World Cultural and Natural Heritage distinguishes between "cultural heritage" and "natural heritage".  

Certain places may enjoy listing as both sites of cultural and natural heritage. The Convention notes that the "deterioration or disappearance of any item of the cultural or natural heritage constitutes a harmful impoverishment of the heritage of all the nations of the world."  

UNESCO sites are not only sources of national patrimony, but also of human patrimony writ large - they are of "outstanding universal value" from the perspective of history, art, science and conservation.  

The Convention defines “outstanding universal value” as “cultural and/or natural significance which is so exceptional as to transcend national boundaries and to be of common importance for present and future generations of all humanity.” 

UNESCO status can apply not only to a single monument, but individual locations and entire landscapes.

Rosia Montana was a significant source of gold in the Roman Empire. Research suggests that this was an extension of previous Bronze Age mining in the Transylvanian area, known as the “Golden Quadrangle”. Mining continued through the subsequent millennia.  While more archaeological work needs to explore Rosia Montana, it is possible that Roman artifacts across Europe may be wrought from gold exported from Rosia Montana. Indeed, Transylvania appears to have been a source of gold in Europe as early as the 3rd millennium BC and some scholars have suggested that ancient Dacian artifacts of gold may well trace their provenance to this very area.

Is Rosia unique enough?

Is Razvan Theodorescu’s claim accurate that there is nothing unique enough about Rosia Montana to merit UNESCO listing?

There are twenty-four mining sites - including individual mines, mining landscapes and mining towns that are UNESCO Heritage sites.  

They range from salt mines in Poland dating back to the 13th century, a hundred-year-old mining town high in the Andes, German industrial coal mines, to a first Century AD Roman gold mine in Spain. 

There are six cultural criteria for a UNESCO Heritage site listing of which a site must satisfy at least one.

In summary they are

1. A masterpiece of human creative genius.

2. An important interchange of human values including developments in architecture or technology, monumental arts, town-planning or landscape design.

3. A unique or exceptional testimony to a cultural tradition or to a civilization which is living or vanished.

4. An outstanding example of a type of building, architectural or technological ensemble or landscape which illustrates a significant stage in human history.

5. An outstanding example of a traditional human settlement, land-use, or sea-use which is representative of a culture or human interaction with the environment especially when it has become vulnerable under the impact of irreversible change.

6. A settlement directly or tangibly associated with events or living traditions, with ideas, or with beliefs, with artistic and literary works of outstanding universal significance.

Twenty-one of the twenty-four the mining areas with UNESCO Heritage status were found to have satisfied criterion 4. Following criterion 4, the next most frequent criteria for mining sites are 2 (16 mining sites) and 3 (12 mining sites).  

No doubt, the most obvious criteria for UNESCO status for the Golden Quadrangle would be criterion 4.  

Rosia Montana, with its rich, multi-ethnic mining history would seem clearly to be “an outstanding example” of a “technological ensemble or landscape” illustrating significant stages in human history. Like the Roman gold mines at Las Medulas, Spain, they may be seen to be an exemplar of gold mining technologies, be they those of the Roman Empire or of migrant Germans in the Middle Ages.  

Criterion 3 seems a good fit, as they are an excellent testimony to the full ambit of the Roman Empire. The last two criteria may be applicable so, at the very least, Mr. Theodorescu’s claim appears to be specious. 

If Rosia Montana is not unique, it is only because it has some two dozen precedents, which makes it a candidate for a listing. This is the irony of UNESCO.

Whether Rosia Montana should be submitted is a separate question. 

To a certain extent, the fight over whether the Rosia Montana Gold Corporation mining project should go forward or  whether the site should be submitted for UNESCO listing has allowed the latter to be painted as a blunt instrument in the arsenal of those trying to stop the former from happening.

This has arguably obfuscated the notion of whether a World Heritage listing is an important goal in its own right.

Also Romania's Minister of Culture must make the submission for a listing and the present Minister - Daniel Barbu - is against such a move.

Heritage means cash

UNESCO has global brand recognition. Many of the most incredible places on earth are listed, drawing hordes of tourists every year. The Convention was written in the seventies, when the notion of sustainable development was beginning to take a hold in international discourse. The Convention envisions both maintaining existing patrimony and promoting development.

According to Eurostat, among seventeen EU countries, only Romania and Bulgaria had fewer annual visitors in 2012 than citizens. 

From these, Romania has the fewest number of UNESCO sites per capita, while the average number of visitors to a country per site per year is nearly 2.5 million.  

Although it is difficult to draw conclusions here, it is clear that the EU leaders in World Heritage sites are also leaders in tourism.

World Heritage listing does appear to have an impact on tourism and  local economies.  

This is felt at multiple levels of a country’s economy, from local direct cash expenditures to state-level taxes.  

According to congressional testimony by the United States National Parks Service, the agency found that “World Heritage designation appears to be economically beneficial and a lure for foreign tourists. It correlates closely with increased visitation.” 

One study in Australia concluded only that World Heritage sites did receive large numbers of visitors and that many of these visitors were foreign.  

One interesting finding in the Austrialian study was that significant increases in visitations were tied closely to major environmental controversies.  

Rosia Montana is such a controversy and it is likely that the present protest - which has amassed 10,000s on the streets of the major Romanian cities of Bucharest and Cluj-Napoca - has received more international attention than Romania as a whole has received in years. 

Ironically, it would appear that if the gold mine does not go forward, the current conflict may well be sowing the seeds for a strong growth in Romanian tourism. 

UNESCO World Heritage status would, of course, be only a part the picture, but it appears to neither be as absurd as Mr. Theodorescu has opined, nor merely a means of halting the gold mining project.