After the biggest data leak in history, we were pleased to announce (only 48 hours later) that we had successfully integrated a source into our Arachnys platform which searches over 200,000 offshore entities and associated individuals which were listed in the ICIJ’s Panama Papers database. This data was based on a selection of 11m documents leaked from the Panamanian law firm Mossack Fonseca.
Everyone hears about the data, the information, and the technical jargon, but we want to give a first hand account of how the data came to be searchable within our platform. So, we found out from an expert, the steps and process which took place to successfully integrate this data into our platform.
An interview with the man behind the data
To explore what it was like from the point of view of one of our tech team who was in charge of integrating the data directly into our platform, we spoke to Tim, who was heading up ‘Project Panama’, as we called it here in the office. Tim is one of the developers responsible for our D3 product.
So, my role in this transaction of data was pretty big. The pressure was on to get the data integrated as soon as possible and with all technical processes, we hit some bumps in the road. Here’s a little insight into how we overcame the challenges, time constraints and issues with the way in which the data was presented to us.
Data from ICIJ
When ICIJ announced the release of Panama Paper data we knew that this was something our clients would be very interested in, and that they would want to use as soon as possible. When the data hit the ICIJ website on May 9th we made sure we that we were ready to start integrating it to our platform.
Analyzing the data
Importing the Panama Paper data wasn’t as easy as it first appeared. One of the most interesting features of the data was the links between different entities. Any company can have multiple officers or addresses attached to it. This gives our clients and their analysts valuable information they can use to pursue entities around the world. We wanted to make these links easily available and searchable, but that wasn’t going to be easy.
Integrating the data
Our tech team had two wildly different ideas on how to approach the problem of importing the data. We were sure we weren’t going to be slowed down by old technology, but instead use the best technology available. With two ideas in the running, most large companies would have picked one and put a massive team on it. They would have taken weeks to get the data up and running. It worked in our favour to be able to dedicate just a few people on figuring out the integration process. There was a real sense of urgency with this information and it was great to be able to provide it to our clients in under 52 hours.
Two of our developers were going to attack both ideas in parallel, so if something went wrong with one we could still have another to rely on. They had the full support of the entire tech team, but were both ultimately responsible for the integration. This meant that if, for some reason, one idea didn’t work we always had the other to fall back on.
Having two people work in parallel on different ideas was a great strategy. After one day we had working prototypes, which our research team could use to test the quality of the data. On day two our research team was testing the prototypes and our tech team kept improving them. They fine-tuned the data and improved the search.
We were then confident enough to push one of the solutions to a secret server that was only available to Arachnys staff. After some more testing there we integrated it on to our application and alerted our clients via email who were available to search the data immediately.
Was the integration a success?
From our point of view it was a major success. The data was released on May 9th and we had integrated it into our Arachnys platform by May 12th. In the first two hours of the data being available on our platform, our clients had carried out over 306 searches; rising to over 1000 searches by the end of the week. Over 1,500 Panama searches have been completed on our system so far. Many of our clients are using it to uncover valuable data and which add real value to their investigation processes.