For a SaaS company founded in the past decade, a US-based hyperscaler offering services for hosting infrastructure would have been the natural place to start. However, the legal landscape has changed a lot, and deciding where to host your infrastructure is now more than just a technical exercise.

In this episode of Sovereign DBaaS Decoded, we are joined by Jens Alm, the founder & CEO of Prorenata, a SaaS company that helps schools document medical records. Jens and our host Vinay Joosery discuss the laws and regulations healthcare service providers must adhere to. They also discuss the pros and cons of hosting your infrastructure on AWS.

Since Jens has moved his infrastructure to a smaller local platform, we asked him to share why he made this move, the challenges he faced, and the lessons he’s learned. He also emphasizes that working with smaller service providers allows for meaningful long-term partnerships beyond customer-vendor relationships.

Episode Highlights

Laws and Regulations That Prorenata Abides By 

”We provide a medical record system, and there are several regulations both on the EU level and in Swedish law that regulate what you can do with patient data, how you need to store it, the security, et cetera.

And some of these are the same for anything like GDPR and all the regulations around that. And others, more specifically for medical stuff. The medical regulations in Sweden are, for instance, the patient data law that gives a lot of restrictions on how, what, who can access the data, and what they’re allowed to do with it.

On top of that, many of our customers have added their interpretations or thoughts and then requirements on how we are supposed to use their data and store their data. And in addition to this, a medical record system is typically a medical device, and a medical device has its whole set of regulations. […]

So there are a lot of requirements, but those are legal requirements that are put on anything from pacemakers to medical record systems in schools. That’s also quite a bit of regulation around documentation. […]

On top of that, we have the fact that we are serving public agencies, and they have different levels of secrecy or publicity.”

So that’s also something that we need to cater to. It’s typically the agency’s issue. But we need to ensure that our product makes it easy for them to say what of our material is public, what is secret, and how we classify it. That goes more into the business logic of our application.”

Can You Remain Compliant If You’re Hosting EU Citizen Healthcare Data on a US Hyperscaler?

”The answer to that is no. We don’t think you can remain compliant if you’re hosting healthcare data on the US hyperscaler. That’s what we think, and that’s what many of our customers think. […]

There’s no final legal statement on that because that seems not to be done yet. But everything points to personal data on a US hyperscaler being an issue. And the more sensitive the data, the bigger the issue.

And we deal with some of the most sensitive data, which means it’s a categorical no to store that on the US hyperscaler unless you make serious commitments to encryption. Or it’s a no with an asterisk — and a categorical no for us — but it’s going to be up to each business and public sector to decide. But people I talk to point out that you should keep it within the EU to be compliant.”

Challenges of Migrating From One Cloud Platform to Another

”We had a lot of issues in the beginning after pulling off the migration. We had to scale because we weren’t able to test the production load in the new environment. […]

We did not know how much overhead Kubernetes adds.

Also, what’s the difference between the MySQL server that we’re going to use now that runs Vanilla MySQL versus MySQL Aurora? Both have more performance but different performance characteristics, mostly in terms of scalability. […]

The clock was ticking. We had customers saying, ‘If you don’t do this by this date, you’re going to be out of compliance’ and not saying that they would fine us but hinting that this was not okay.

We had files in multiple locations. We started with only moving the files for the customers that were hard on the date — and then keeping the massive terabytes of files that took months to move in an orderly fashion over to the new provider because they also had capacity issues.

We were used to being at AWS, and if we need capacity, you say click, click, click, and you got it at our scale. But when we moved to a much smaller Swedish provider — a company approximately our size — they might not have the capacity.”

A Silver Lining

”We got up on Kubernetes, and we’re super happy about that. We can now spin up a new environment in minutes for testing, QA, and so on. We can have customers spin up a sandbox environment if they want to integrate with our API; they can get their sandbox. That was pretty hard before, and we can do it now easily.

And we also deploy much more easily and consistently. We can build the Docker images on each push of code, et cetera, et cetera. And we can also switch providers because we are working on top of generic open-source frameworks like Kubernetes and Docker.”

Here’s the full transcript:

Vinay Joosery: Hello and welcome to Sovereign DBaaS Decoded. I’m Vinay Joosery and this episode is brought to you by Severalnines. We provide enterprise automation software that orchestrates high availability open source database operations in any environment while maintaining total control. Our guest today is Jens Alm, CEO of Prorenata in Sweden. Thanks for joining us today, Jens.

Jens Alm: Thanks for having me.

Vinay Joosery: Can you tell us a bit about you, who you are and what you do?

Jens Alm: Right, yeah. My name is Jens Alm. I’m the CEO and founder of Prorenata, a company that makes records for schools, software as a service. I’m a medical doctor by training. I’m also half a software engineer. Never finished that, but kind of returned to it. And I’m married and I have three kids of various ages. And that’s about it, about me. Yeah.

Vinay Joosery: Okay, great, great. And Prorenata is doing software around, let’s say, medical records.

Jens Alm: Yeah. So we’re doing medical records for schools, which is kind of… a niche business to business case. And also we’re very particular for Sweden, like that’s Sweden is where we’re situated, and it’s also our only market. And this is for school nurses, for them documenting, for instance, vaccinations and measurements of students, but also for the wider array of student health care. Which could be things like bullying cases or special needs of any kind, and then the documentation and case management support for those cases. That’s what we do when we got approximately 50% of the Swedish market doing that. So we are in many different schools all over Sweden, including the schools of my kids, actually.

Vinay Joosery: Okay. Yeah, yeah. All right. And it’s a SaaS platform that you’re building.

Jens Alm: Yeah, it’s a SaaS platform. I guess we’re going to dive deeper into that. But it’s a pretty standard SaaS platform, using pretty standard web technologies and lots of standard backend stuff. And we’ve been around for approximately 10 to 12 years. So we’ve seen a few different shifts in that platform. But the general strategy is still software as a service. We handle everything. Nothing that is on-prem or anything like that for the customer. But you go to a login page, you log in, we have the rest.

Vinay Joosery: Yeah, yeah. All right, all right. So I guess for any SaaS company founded in the past decade or so, a U.S. hyperscaler would have been the natural place to start off. So hyperscalers, they offer compelling services for hosting infrastructure, But the legal landscape has changed a lot. So we have data protection regulations like the GDPR. We also have contractual obligations towards customers and suppliers, right? So deciding where to host your infrastructure is not just a technical exercise anymore, right? So can you tell us a little bit about Prorenata’s infrastructure journey?

Jens Alm: Yeah, absolutely. So to do that, I have to take just a short detour and talk about like Prorenata’s journey as such, because they are obviously interconnected. And we started very, very small. And that means we were two users to start with. Like, the software that I built in the beginning 12 years ago was for me as a medical doctor on a school healthcare in one school. And for my nurse, like the nurse that I worked with. And those were the first two users for the first, like six to nine months or something of the product. And back then, it wasn’t really a product, it was a project of mine, like at nights and weekends.

And obviously the infrastructure back then was a virtual private server somewhere in Europe, I think, hosting a Django app and a MySQL server. Basically on the same server. And that kind of held up for quite some time, actually. For at least a year or two, we had a similar solution where we used like a rackspace server and just like upgraded the server capacity of that one server. That hosted everything for us as we grew. Because in the beginning, like growing, was from going from two to 15 users, and then to 30 users. And so on. So the scale was still very small. And then, approximately 10 years ago, we shifted to Amazon and AWS. So doing that meant that we could start working on like using the S3, for instance, file storage. Before that, we actually stored files in the database. Not ideal.

And after that, we moved into AWS, and we had several AWS services and we started to actually explore. And that has been the path for the like… up to approximately three, four years ago, we were fully AWS, and we use several AWS services. We used the EC2, obviously S3, but also RDS. And first I hosted MySQL and then actually the AWS Aurora and their MySQL equivalents. So we were very happy doing that. And then came several legal issues that we had to deal with. And I guess we’ll get into that. But that meant moving away from AWS and looking for Swedish alternatives, European alternatives, but in particular Swedish alternatives. And that’s what we did.

And I don’t know if we’ll get into that more later, but that meant looking for services that could replace both AWS, but also the services we use on top of AWS. Or like the higher-level services of AWS, like RDs and stuff like that. So both like EC2 style services and just getting hosts, but also the fact that we, for instance, did not have a database administrator. And using backups, like adding backups to our service, was clicking on a checkbox. And we wanted to keep it, if not exactly that way, at least like to that level, because we didn’t have the capacity to do more advanced database administration and so on. We wanted to focus on the business part of our product.

Vinay Joosery: Right, right. So I guess there was something around, you know, something must have happened then when you were moving from Amazon, right? Which made you decide to move from Amazon. But I guess, you know, healthcare is a regulated industry, right? It’s very well known to be regulated with its own data privacy laws, perhaps even to a higher standard than the GDPR. So can you tell us a bit about the regulations that your data is subject to?

Jens Alm: Yeah, and that actually cuts directly into that and that decision. So we provide a medical record system, and there are several regulations both on the EU level and the Swedish law that regulates like what you can do with patient data, how you need to store it, the security, etc. And some of these are the same that goes for anything like GDPR, for instance, and all the regulations around that. And there is more specific for medical stuff, and the medical regulations in Sweden is for the patient data law that gives a lot of restrictions on who can access the data and what they’re allowed to do with it.

On top of that, many of our customers have added their own interpretations or thoughts, and then requirements on how we are supposed to use their data or store their data. And that can be regulated in different agreements with them. And that typically points to a standard like ISO 27001 or something like that. And in addition to this, a medical record system is typically a medical device. And a medical device has its whole set of regulations in the CE marking. This is something that is encapsulated in the MDR, or now the MDD directives in European law. Like, there’s a lot there, there’s a whole lot in there, and it’s actually changing right now, so there’s a lot of requirements. But those are legal requirements that are put on anything from pacemakers to medical record systems in schools.

So that’s also quite a bit of regulations around documentation. We need to prove that our system has mitigations in place for different risks that are medical. And those could be things like, if we’re showing a growth chart on a patient and we are doing some calculations or showing some reference curves, that is diagnostic information for the school nurse. So he or she is making a diagnostic decision based on the calculations we make in the system.

We then need to prove that those calculations are medically correct. And we live on the kind of outer edge of that regulation. Because that regulation, as I said, also focuses on pacemakers and other stuff you put in your body, but also like plastic gloves. So like, it covers the whole spectrum and we live a bit on the outer edge because we do so little of our stuff. It’s very medical and it’s not like we don’t do hospital systems, but it still very much affects us. And our customers expect us to and require from us to fulfill these requirements from the law.

So, yeah, we have GDPR privacy, obviously. We have information security connected to that. And then we have the medical, the fact that we store medical information. So lots of stuff to keep track of. And yeah, on top of that, we have the fact that we’re serving public agencies and they have different levels of secrecy or publicity, like privacy. So that’s also something that we need to cater to. That’s typically their issue, not ours, but we need to make sure that our product makes it easy for them to say what of our material is public, what of our material is secret, how do we classify it, and so on. That goes more into the business logic of our application.

Vinay Joosery: Yeah, yeah. Okay, okay. So I guess all this regulation on the data, it means pretty much that you have to… companies in the industry, they have to adhere to them. And if they don’t adhere to them, then you would expect fines, right? So have you seen fines being given to other providers in the healthcare industry?

Jens Alm: Yeah, actually, yes. So there are several Swedish agencies that are working on this. So connected to the GDPR, we haven’t seen it in Sweden, the healthcare system. I don’t think the GDPR privacy boards have reached that area yet, but we’ve seen it connected to the other regulations that I talked about, specifically about patient data and integrity around patient data. We’ve been involved as just a supplier of a system when our clients have been approached by the Medical Board of Sweden, and they’ve been looking at, or the Board of Integrity in Sweden. And they’ve been looking at different things about how they set up their different secrecy and privacy regulations. And there were fines, not against any of our customers, fortunately, but there were fines levied in those cases.

Like a year or two ago for several Swedish public and private health care providers. Because they did not provide sufficient privacy and sufficiently well-documented routines around who gets access to patient data. So that’s typically, they need to be able to show that they can control, that they log everything, that they check the logs, that they follow up on whatever findings they find in the logs, and about what users access what data. And that also that they have reasonable protections in place so that patient data doesn’t go to someone they shouldn’t. And that is actually within the organization. That’s not about data breaches. That is about putting up the right checks inside the organization. And there were several fines in the order of hundreds of thousands of dollars, or millions of Swedish crowns levied to those different organizations.

Vinay Joosery: Right. Right. So, you know, the fact that there’s not that many GDPR fines, I mean, especially in healthcare, I mean, does it mean the GDPR has no teeth? Or is it that the authorities, at least the GDPR authorities, they’re not really auditing and ensuring that companies comply? Or maybe because it’s healthcare, you know, there is another body that will actually… I mean, you already have another watchdog, right?

Jens Alm: Right. So the watchdog that did… like do those things, and levied the fines, they were the data privacy watchdog in Sweden, like the agency concerned with these things. But they didn’t use the GDPR legislation when doing that because they look at many different legislations. And the GDPR legislation does apply to healthcare, but there are several other laws that kind of supersedes it in several areas, which means that it gets more complicated than just GDPR. Which means that I do think that GDPR will not be used necessarily as like a major area in the healthcare industry.

Because there are so many other privacy protecting laws already that cover it. And GDPR basically cover the things that they don’t. And mostly, there are things about the way GDPR is structured, you need to have a legal reason for storing the data. And typically, one of the legal reasons is that you are obliged to store the data because of some other law. And the way healthcare works is you are obliged to store the data for patients that you treat, which means that there’s almost always reasonable costs to store the data, according to GDPR.

So, even though GDPR does apply to healthcare, it’s typically kind of an open-shut case on the GDPR level where other areas are more concerned. So it’s not that they are concerned with you are not allowed to have this data, which could be a GDPR thing. It’s more like, when you have this data, have you treated it correctly inside your organization? Have you put up the correct boundaries inside your organization about who in your organization gets access to this data, which is typically not a GDPR issue. It’s typically about the other areas of legislation.

Vinay Joosery: Right. So now comes the million-dollar question. Can you remain compliant if you’re hosting EU citizen healthcare data on a U.S. hyperscaler?

Jens Alm: Yeah. So, we said we were going to come into this basically, because why did we move? We were actually pretty happy on AWS and we had other fish to fry. But we had to take a pretty major, like make major investments to move to another provider. And the reason was that we think that the answer to that is no, we don’t think you can remain compliant if you’re hosting healthcare data on a U.S. hyperscaler. Now, that’s what we think. And that’s what many of our customers think. And what they let us know, in no uncertain terms, that they think. We’re not… I can’t make a final… like, there’s not a final legal statement on that, because that seems to not be done yet.

But everything points to personal data in a U.S. hyperscaler at all is an issue. And the more sensitive the data, the bigger the issue. And we deal with some of the most sensitive data, which means that… I would think that it’s a categorical no to store that on a U.S. hyperscaler unless you make serious commitments to things like encryption and stuff like that. And you really know what you’re doing and who has the keys, and where is it encrypted and decrypted, etc. And doing that really hampers the actual use of any such hyperscaler. Because if they can’t see your data, it becomes harder to use their services, obviously.

So I think it’s a no, but I think it’s a no with an asterisk. And there are some… it’s a categorical no for us, but it’s going to be up to each business and public sector, etc. But everything… people I talk to point to, you should probably keep it within the EU to be compliant. And that’s the conclusion we came to, in like a couple of years ago when we decided to move. We had started to get signals from our customers, some of our larger customers and some of our prospective customers as well, which are some of the most important customers, the ones that were looking at switching to us, but they were saying that we probably can’t do this unless you solve this issue. The issue being that you have data… we had data in the EU, but on AWS.

We then decided that, yep, we don’t… even though the jury was still out a bit, we did not want to have these discussions. Like, these were not the discussions we wanted to have with our customers. We wanted to discuss features and safety and everything else. We did not want to discuss where our data is hosted. We wanted that to be a non-issue. And we thought that this could really be… go from being something we had to apologize for all the time to actually being a business advantage for us and kind of a moat really in like business and startup terms. That we were going from like having a problem, having to address this and having to really make sure to talk about encryption and stuff like that, to saying, yeah, that’s a non-issue.

And then we can move on and talk about other things. And also be kind of certain that any other business entering our market, any potential competitor, would have to comply with the same things and actually try to educate the market, in our case, because we are one of just a few players in this market. So what we say in this little niche market will probably spread. So if we try to educate the market that the kind of Apple way of doing things is… you say that you don’t need it until you do it. And then you say, this is the only way. And we kind of went down that road. We first said about a year that we don’t see this as an issue. And then pressure mounted. And we kind of caved to that first. And then we kind of realized, or I realized, that this is not only… We should actually go all in here. Like if you do it, we should do it for real. And not only EU, but then we put it in Sweden. Not for legal reasons, actually, but for marketing reasons. We wanted to be in Sweden. We could have been in EU, but we wanted to be in Sweden.

But also for operations reasons. It’s good for us. We’re a Swedish local company. We don’t have any plans for anything else right now, which means that having a Swedish partner makes it easier for us. Just having a Stockholm-based partner for us makes it easier. But it’s also very much easier… it’s an easier message to our customers, who are all Swedish-based. And public sector, that it’s not in Germany, even though that would be legally fine. And then we can move even faster beyond the discussion of… to more interesting business discussions. Yeah, so that was the reasoning. And it came first, hesitantly. And then once we pivoted, we tried to do it as quickly as possible and go all in so that we could like get an advantage up instead.

Vinay Joosery: Yeah, yeah. So I guess that’s moving off AWS. And even though it’s in Europe, because sometimes they do say, well, we have data centers in Europe, but the actual localization of the data, I mean, where the data is actually stored, is one thing, but the actual jurisdiction is a different one. So you needed to have EU jurisdiction on the data.

Jens Alm: Yeah, so we already had the data physically in the EU, on AWS. But that obviously did not suffice for the legal requirements.

Vinay Joosery: No. And I mean, you did kind of touch on that, encryption, anonymization, pseudonymization, right? These are things that maybe could be used to sort of somehow get around having clear data on the cloud. But these could perhaps get you compliant staying on AWS.

Jens Alm: Right, but they could have. But we explored it, not very optimistically, but we wanted to make sure that we were making the right decision because it was a huge decision to make. So we explored that option. First of all, it would have been a huge engineering effort to get that encryption in place because we were using AWS heavily. And we’re using their services, like their high-level services, like RDS and so on. And making sure that… all personal identifiable data is encrypted, while still using their high-level services would have been almost impossible. And making sure even if we started just going down to EC2 and just using virtual servers again, which kind of defeats the point actually of using AWS in our case, then we can move to anyone, that would still have made it… like make sure, really sure, that nothing ends up in a log anywhere that could be personally identified. Not IP numbers of our customers, not anything. And that becomes really hard.

So that was first, like engineering-wise, we probably wouldn’t be able to make it. And the second part is, even if we did make it, it would have been a very complex solution. And since it is a very complex solution, explaining it to our customers and why this is reasonable, and why we… and the difference between pseudonymizations and anonymizations, and who has the encryption keys, and why it’s okay that they have one key, but we have the other or something like that, it would have been really hard technical discussion to have that we like wanted to get that out of the way. That’s one of the points.

We didn’t want to have those discussions, but yeah. No, we decided pretty early that that was a non-starter. We wouldn’t be able to do that for our workload because our workload is so complex. We have huge data sets and very complex relational data and so on. But if we were to store only files, if that was it, we could probably get away with it. We could probably store files in S3, like encrypt them, store them, and then decrypt them when you get them up. And that is something we could look into. But again, if we were to do that, we would still have… like AWS would still show up in a sub processor list in all the agreements we would have to send to the municipalities. And I would have to take personal calls on each and every one of them to explain why that was okay, probably. And so from a business standpoint, that wasn’t really a good idea. So no, for us, no, that wasn’t really meaningful.

Vinay Joosery: All right, all right. Great, great. So moving on a bit, how did you plan this emigration? What were your requirements on the new infrastructure environment?

Jens Alm: We planned it slowly and then suddenly. The requirements we had, first of all, we have a pretty big data set, both on the database side and stored files. So database files in the hundreds of gigabytes back then, larger now, and tens of terabytes of files. So not massive by cloud scale, but huge enough for anything to be complicated, because just transferring things takes hours and stuff like that. So we had to plan for that and we couldn’t afford more than like taking 24 hours of downtime, that was probably the largest downtime we could sell to our customers without it being a massive inconvenience for them. Pretty much a nine to five workload, typically, like Monday through Friday, maybe seven to six, to be honest, work hours. So that’s okay.

But we had to plan it to be able to do it in 24 hours. We set up actually with a different provider than the one we have now, for different reasons, but with a local Swedish provider. And we also decided at the same time, which in hindsight, might have been a bad idea, but at the same time moved to Kubernetes. So moving from what we had, which was basically carefully tendered servers that we logged into and ran Bash scripts on to do updates and did Git pull and stuff like that to do the updates. From having that to saying, no, we’re going to containerize our application, we are going to run it on Kubernetes, and we’re going to make it stateless so that the stuff that runs on Kubernetes is fully stateless and can be just reloaded and so on. And we did succeed on that, but it was a pretty big undertaking. And we containerized a Django app that is in the hundreds of thousands of lines of code.

And so it’s a big monolith. Really doesn’t lend itself that well to containerization. Each instance of it takes gigabytes of memory to run, et cetera. But it works. And we’re happy we did it, but I’m not happy that we did it in that way. Because the problem was that we had so many technical issues to fight with in learning Kubernetes and setting up Kubernetes that we did not have time to do enough testing of the actual migration. And we had quite a lot of issues in the beginning after pulling off the migration. And we had like scaling and so on. Because just moving the environment, we weren’t able to test production load in the new environment.

So we kind of had to do a like just educated guess on how much server do we need, etc. Which is also much harder because we at the same time containerized the application. So we did not know how much overhead does Kubernetes have, if any? How much overhead, like, what’s the difference between the MySQL server that we’re going to use now that runs Vanilla MySQL versus the MYSQL Aurora that both is more performant, but also has different performance characteristics, mostly in terms of scalability. That it typically might not be faster on the single query, but it scales better. For multiple queries or complex queries. And we did not know. We didn’t have real time to find out because the clock was ticking, because we had customers saying that, “If you don’t do this by this date, you’re going to be out of compliance.”

And not saying that they would fine us, but hinting that this was not okay. So we kind of had a pretty hard date as well. But we did pull it off… with some hiccups, and we planned it so that we could do it within 24 hours. Basically, a huge MySQL dump and get it up again. And that was a big thing and having the files in multiple locations. We actually had files in multiple… we started with not moving the files, but only moving the files for the customers that were mostly hard on the date. And then keeping the massive terabytes of files, like that took months to move that in an orderly fashion over to the new provider because they also had capacity issues. Because obviously we were used to being at AWS. And AWS, we’re a very, very small fish in a big pond. And that meant that if we need capacity, you just say, click, click, click, and you got it, at our scale.

But when we moved to a much smaller Swedish provider, actually a company approximately our size, which at the time was like 25 people or something like that, then they might not have the capacity. Like when we say we want 50 terabytes of disk, and we would like to have actually times three during a transition period because we want to have the multiple places for a while and so on. They say, “Yeah, no, we have 75.” And like, okay. Okay, then we hold off for a couple of weeks until you order more discs and then you had like disc issues, like delivery issues. And we were not used to that.

At our scale, obviously S3 can take anything you can throw at it and we don’t have to worry about it. And the abstraction holds that it’s this magic thing where you just stuff things and then you get them back. And now we have to think that, oh, no, there’s actually a hard drive somewhere. It’s still somewhere underneath all of this, and that has to exist and be delivered by a company and so on, installed and tested. But just like the silver lining of this, we switched the provider again for different reasons, but the silver lining of this is we got up on Kubernetes.

We’re super happy about that. We can now spin up a new environment in minutes for testing and QA, and so on. We can have customers spin up a sandbox environment for their… if they want to integrate with our API, they can get their own sandbox. Stuff like that was actually pretty hard before, we can do now super easily. And we also deploy much more easily, much more consistently. We can build the Docker images on each push of code, et cetera, et cetera. And we can also switch provider because we are working on top of generic and open source frameworks like Kubernetes and Docker and images and so on, and OpenStack. And if we switch provider, we can just do that.

And I don’t know if this matters, we are in the Severalnines podcast, but we’re also using cluster control. And very happy about that, because that kind of gave us some back of what RDS gave us in terms of getting an interface, getting a UI, getting some automation for stuff like backup management, etc. And it made it so that we still don’t need to hire a DBA just to do those things. So that was really good. But we had to find all those tools as well. And tools on top of Kubernetes to work with. There are a million tools on top of Kubernetes. We had to find which ones are good, etc., etc.

Vinay Joosery: So Kubernetes was a key platform. And then you moved to OpenStack, which I guess you didn’t pick the other… you know, the other one is VMware that could potentially be used by…

Jens Alm: Yeah, we were looking at it. It was more about what providers were… we’re looking at providers first and Stack second, but pretty quickly we’ve found that the decision I wanted to make is if we move again, I want us to be able to move again if needed. I don’t want us to be in the situation that we were at where it takes a year. It took somewhere between 9 and 12 months to pull this off. And we had to do massive resources to do that. And I wanted to be able to do that again, much more quickly, if needed. So I said, basically, rather OpenStack than VMware, because OpenStack is open. I’d rather be in the open source. We’re pretty used to working in an open source environment, using Django and Python and stuff. MySQL and stuff like that.

So that came pretty naturally, but was mostly decided by selecting what company we wanted to work as a provider. Which happens to be a company now called Elastx that we’re really happy with, and they use OpenStack. It was also easy to then use your own Terraform files and like kind of keep it in control. We did not at the time want to use a Kubernetes as a service. First, because that was still early days for that. And it felt like the ones that did provide a Kubernetes service were pretty limited. And second, because we felt like, or I felt like, if we wanted to go into Kubernetes, we need to know it. We need to know what we’re doing.

So I wanted us to like feel the pain and not hide it away and then be surprised. And I think that was the right decision, because even though we’re actually now moving to a managed Kubernetes solution on top of OpenStack, we are at a place where we have knowledge inside the company on how to handle different Kubernetes issues and deploy. But VMWare didn’t really show up, so we didn’t do proper evaluation.

Vinay Joosery: Yeah, yeah. Okay, okay. That sounds good. I mean, I guess that was a very painful process. And what I can appreciate here is you actually used that opportunity to make, in a way, yourself more resilient and independent from the underlying infrastructure. So that you decoupled yourself from a single vendor’s infrastructure so that you could move around if needed. And that’s a tough thing, but that’s absolutely the right way to go. Because it means you are in a way, you are less reliant on vendors, and you’re future-proofing a bit the business and the infrastructure.

Jens Alm: Yeah, and we actually benefited from that just six months in. Because we selected one provider. And they had scaling issues and some pretty major… in their OpenStack solution, had some pretty major stability issues. And so after three or four months, we basically said, this is not working. We’re having weekly downtimes. Our customers are not happy. So we need to move. So we found another provider. We checked with them first. Do you have the same issues? No, we don’t. Trusted them. And we were able to pull that move off in just a month because we were now not dependent on one single vendor or locked into a single vendor. And because we had our own Terraform files, that could get the Kubernetes up and running and everything on. Just give us the OpenStack keys and we can do it ourselves. And that was really good to be able to do that. So it paid off actually immediately. So we’re really happy about that.

And it also makes it easier for us now to move to a managed Kubernetes solution a couple of years later and say that, yeah, we want to move up the stack now a bit. And we feel confident that the managed Kubernetes solutions have matured. And we want someone else to deal with that part for us. Because of the way we built it and the layered way we built it, we can do that. It’s actually not a huge undertaking, it will take a month or two to validate everything and so on, but it’s not a massive undertaking.

So yeah, I’m really happy with the end results, even though I think we could have done… made it easier for us by moving first to Kubernetes and then leaving Amazon or the other way around. Yeah, I think doing both at the same time made it much harder to… too many moving parts. We should probably have containerized on top of Amazon first, I think, because it kept RDS and stuff like that, and then be prepared to move it all to somewhere else. But that’s hindsight.

Vinay Joosery: Was that the hardest part?

Jens Alm: The containerization? That was hard, but I don’t know. It was the whole thing. The problem is like, as soon as there was one thing, that, together with the scale of things, so that if we had a failed migration, it took three days to make a fix and then try again. Because we realized after moving 70 terabytes of data that it didn’t work or something. So I think we should have been more orderly and tried to do one thing at a time. Because then it’s easier to isolate errors. And learn. So I do think that Kubernetes was hard.

The hardest part was the database, where we unknowingly upgraded from MySQL 5.7 to MySQL 8.8 because of a faulty configuration, which also bit us a bit in the end. Just because there were too many things moving. And someone pulled in a MySQL image and got :latest instead of :5.7, and no one really realized. And it worked until we started to hitting the minor differences. or places where our database included some kind of not super validated or almost corrupt data that MySQL 5.7 thought was fine, but 8 kind of stopped on. So different things, but too many of them at the same time. I think that’s the lesson. We would probably have done more, like, ran into most of these problems anyway, but we would have taken them one at a time. And now it became a bit frantic.

Vinay Joosery: Right, right. Yeah. It works until it doesn’t.

Jens Alm: That was the thing. And we thought that more worked than did once we hit migration day. And we fixed things and it was okay. But that, together with the fact that the underlying infrastructure was actually a bit shaky, was very frustrating. Because it was network issues, which is kind of the worst issues, because you don’t know what’s happening. And if you then have Kubernetes abstraction layers on top of that, the one thing we did not manage to get in place in time was proper monitoring. So at Kubernetes, which we were unfamiliar with, which abstracts away a lot of the network stuff. And then we had that on top of OpenStack, which we were unfamiliar with. And then we get random network issues, like four minutes here, four minutes there. Things got disconnected. Services got disconnected from each other.

And yeah, that was a period of my life that I do not wish to repeat. We’re trying to find those things until we realized that this is actually in the infrastructure layer. Because there were so many things we’ve changed. Like, really finding… And you don’t want to blame the infrastructure, because no, it’s probably our helm chart, or it’s probably… or it’s this, or it’s whatever… the nodes network, or the something, something, something. Kubernetes live in this probe, something. But, no, it was the network. There was obviously other things as well. All of those things that I just mentioned, were also problematic. That was the problem. Like we had several things. Yeah, too many things.

Vinay Joosery: Okay. So, you know, I guess you’ve run both on AWS, right? And now you’ve been running on a more, let’s say, national, smaller cloud, right? So how do you compare running infrastructure on a hyperscale versus, you know, sort of smaller scale local cloud, right? In terms of compliance, we know the answer. But how about, you know, in terms of cost, functionality, support?

Jens Alm: Yeah, so I think cost was initially higher for us, but we were over-provisioned for quite some time. Because we didn’t have time to kind of tweak it and find the right levers. And I think it became… similar after a while, we have a bit lower infrastructure cost, but we have to buy some licenses and stuff on top of that to replicate some of the functionality on AWs that we use. So I think we have similar costs approximately. So the good parts about using a smaller Swedish local public cloud is that we have a really good relationship with what is our most important provider, is like, this is our core business relationship with any supplier, this is the one that is most important, that it works, and it really does, and we really have, on a like personal level, a good relationship with this provider.

And that is obviously not something we had on AWS. It’s like, we are not a big company. Especially back then we weren’t a big company and obviously AWS scale, we’re obviously absolutely not a big company. I think we had some kind of account manager in the end, none that I actually had… and I had a name, but that was about it. But mostly you just have the API and you have the UI and that’s it, and that is your interaction with this company. That and the bill, is the one thing, the one time, the one time you actually get contacted by AWS is when the bill bounces. Then they’re quick and then they contact you. But otherwise, like getting support, et cetera, we had some issues that sometimes several years ago were really hard to get to. So that is obviously much better now, the relationship and so on.

On the other hand, we didn’t need to have a relationship with AWS typically, because when I say that, though, our only relationship was through the API and the dashboard. That was typically the only relationship we needed to have. With our new provider, we have talked about them, about instance flavors, such as, you have one with this many CPUs and this many gigabytes. Could we get one that doesn’t really track with our load? We would like more memory, fewer CPUs, or something like that. With AWS, we don’t need to have that conversation. We just choose another instance flavor, and obviously they have them.

Also, things that are more mature, like on encryption level, like all disks are encrypted, etc., etc. We had to kind of move, get a thing called Orchesto by a company called Zebware to encrypt our files. Because the swift storage, the S3 like storage that our provider has, did not have disk level encryption. And we promised to store everything encrypted at rest. So we had to encrypt it ourselves, et cetera. So there were several things that we had to kind of fill in the gaps that we wouldn’t have had to in AWS. Because it’s more mature, and it’s also the number of services provided is just massive. And with AWS, you could also make experiments with different new services and say, oh, maybe we can use this cache thing here and we can try things out. And now, if you do that, you kind of have to deploy it yourself first, which Kubernetes makes it easier to do that. But it’s still not as easy as just click, click.

But in general, it’s a net positive. It’s definitely net positive. Having the relationship is definitely net positive. And if it wasn’t, it wouldn’t matter, because compliance reasons means that we would have to do this anyway. At our size, it’s nice to have a company that is approximately also our size. And that you have the empathy towards each other on a human relational level. And that really matters, actually, turns out, with one of our most important business relationships that we have, to actually have another person that knows who you are and kind of feels your pain in the other end. And vice versa that it’s nice to talk… like when they say, “You’re doing this and that with our API. Could you stop doing that because it’s really a performance issue?”

Okay, sure, we can try to do it like you’re supposed to do it this way. Could you do it that way instead? And we can rewrite our app to do it that way or whatever. So it’s a vice versa. It’s an actual communication. It’s not like I feel like Amazon would ask us not to use it in a particular way. We wouldn’t be able to. They would have safeguards in place and stuff and rate limits and whatever. But it’s good to have a personal relationship there. Right, yeah. Business is local in a way. Human relations matter. Yeah, human relations matter. I think that’s… and hyperscalers don’t deal in human relations because they don’t scale or don’t hyperscale, at least. So, yeah, but especially if you’re a small company, I think for everyone, but for a small company, it outweighs other areas.

Vinay Joosery: Yeah, yeah. All right. So I guess there’s probably many other companies in the same boat, right? Probably started in the last 10 years or so, running on a U.S. hyperscaler cloud vendor, right? And probably being non-compliant with privacy laws, right? So what are your tips for those companies?

Jens Alm: So, everyone has to make their own list. Everything comes down to a business risk in the end. If you have compliance risk or an IT security risk, or anything like that. In the end, it’s a business risk, and you have to make a business decision on how to treat your different business risks. It’s obviously a business risk to not be compliant. It’s a legal risk that becomes a business risk. But it’s also a business risk to spend a lot of time and money on an issue if it doesn’t matter to you in the end, because there are opportunity costs to that. So I believe that being compliant is about reducing business risk.

It’s also about ethics, because maybe you can get away with it, but you’re supposed, especially in the field that deals with public healthcare and so on, to behave in an ethical way. I want to be able to say to my customers that I did the best I could. If we were to have an IT attack, a ransomware attack or something like that, I want to be able to lay all cards on the table and say that you can see that we did all the things we could. This happened anyway, because you can’t eliminate all risk. And I think compliance comes down to the same thing, because in our operations, we’re still using things like Google workspace and stuff. We’re using American companies.

When I send an email or I have a virtual meeting with one of our customers, their IP number, at a minimum, their IP numbers will be stored at the American server. That is something that most companies do, including our customers. And that’s a decision that they made. And that is kind of… is that compliant? Can I ask Max Schrems about that? And he would probably say no. So would probably most of the EU privacy protection boards. But most people are doing it still, and we are still doing it, and most of our customers are. But we’re exploring other options. But that is different from the very sensitive data that we care for for our customers.

And I think that there’s a scale there, as with anything in life. That compliant isn’t just a binary thing, but you’re weighing different business risks, and you’re weighing different legal risks, and you’re weighing different operational risks, et cetera. And you as some kind of management team have to make those decisions. So what it comes down to, for those working in the healthcare sector, I would say, if you’re not exploring this, you’re probably very late to this game. You should at least be checking what your customers ask for.

You should probably have some kind of plan now for how, like if you haven’t moved things to EU, you should probably have a plan for it. And you should probably have a reason, like a canned response to your customers on why you haven’t done it yet. And if you have a reasoning that involves… This will probably blow over, you’re probably on the wrong hand, because it doesn’t seem like it will. So this is specifically healthcare and other like legal… legally regulated areas, I would say that, yeah, data should be in the EU if your customers are in the EU and you are affected by European law. And if you don’t have a plan for that, you should make one pretty quickly.

Vinay Joosery: Not just in the EU, on any cloud, but probably in terms of ownership as well, the company should be owned or should be under the EU.

Jens Alm: Right, it should be… exactly. When I say in the EU, I mean legally in the EU, not just fiscally in the EU, but legally in the EU, so that you are in an EU jurisdiction. And the speed at which you should do that is up to you and your customers, but you should have a plan at least on how to do that. And you should have some kind of explanation on why you haven’t done it already. And then again, many companies haven’t. So you’re in… if not good company, you’re in company, at least. It’s going to be more and more urgent to do these things. And there’s no sign that this will blow over. On the contrary, it’s getting more and more towards that this needs to be fixed.

So there are things you could wait for, as like a third privacy shield, or like a second privacy shield coming, and so on. You could wait for that. If you use sensitive data, I wouldn’t hold my breath on that. I don’t think you’re in good company there. And also use this, my recommendation, use this as a business opportunity, because it definitely is. And it has been for us that do the investment and see it as a business investment to get data sovereignty, because your customers will be more and more interested in having data sovereignty. Kind of regardless of where those international treaties, where they end up. Because even if they end up so that you can do it this way, everybody is kind of burnt on the different legal decisions that have been made throughout the years.

And the fact that you kind of have changed it. And you know that if I put the data in the EU, it’s going to be okay. And if I put the data in the U.S, Max Schrems might come for it anyway. And you’re not going to be safe. And one thing that you’re selling to your customers is they’re able to sleep at night. And your customers being whomever, if they are some kind of CISO or CTO, or somewhere having purchasing decisions, they want to be able to say that they did their part in covering for different legal aspects. So if you can provide them with that solution, you have one up on anyone who needs to have a discussion about encryption and privacy shield and so on. If you don’t need to have that discussion, you can discuss features instead. And say that, we got you covered there. You don’t have to worry about that. Let’s move the discussion to things that are more important for both of us. And that’s something we really enjoy, the fact that we can say that, no, it’s good. We’re in Sweden, everything is Swedish, we know where every bit is, so to speak. And we know that the NSA is probably not looking at it. Not legally, anyway.

Vinay Joosery: Yes. So I guess, for our listeners, Shrems II is the invalidation of the EU-U.S. privacy shield. That was about two and a half years ago. And there’s rumors that there will be maybe a new replacement for the privacy shield. But fundamentally, the U.S. jurisdiction and the jurisdiction in the EU, it’s very different about how we treat personal data. And that’s the big problem. And laws take a long time to change. That’s the big problem. And the best time to move is… it would have been three years ago. The next best time to move is today.

Jens Alm: Yeah, no, absolutely. It’s one of those, yeah. And I think if you look at the geopolitical climate right now, data sovereignty is more important than ever, and will probably be increasingly so, because the old privacy shield deals were made in a very different geopolitical climate. And having the U.S. backing down on surveillance now is, I find, as not a political analyst, but I don’t find that likely. So whatever they give up when trying to sign any kind of privacy agreement, I don’t think it will be enough. Like, that’s my armchair general analysis of it is that I can’t see them in this political climate right now giving up surveillance rights if they don’t have to. So they will give up something because of businesses, because they want people to still use Google and Microsoft and AWS. But I think in the end, it won’t be enough, especially not for sensitive data.

Vinay Joosery: Yeah, or either the EU backs down on the GDPR.

Jens Alm: And that doesn’t seem likely, either.

Vinay Joosery: No.

Jens Alm: Yeah, no, no. That’s not going to change anytime soon. And if that changes, that’s a decade from… because of just the legislation speeds of the EU.

Vinay Joosery: Yeah.

Jens Alm: So, yeah.

Vinay Joosery: Jens, that’s a fantastic discussion. Really appreciate having you here and share your thoughts with us.

Jens Alm: Thank you. Yeah, this was really fun. Really like diving into these areas.

Vinay Joosery: Yeah. So the best description I saw of you was in one of the talks you did at, I think Open Infra Days a couple of months ago; Jens Allen, CEO, medical doctor who was so frustrated by the status of medical software that he built a medical record system and accidentally started a company.

Jens Alm: Yeah, it’s kind of accurate, actually. Yeah, I kind of backed into that one. Yeah.

Vinay Joosery: Yeah. All right, well, thank you very much. And for our listeners, thank you, folks, and see you for the next episode.