My 1 year older brother (to the left), my mother and me (to theright)
My mother has been working for one of the largest banks in the EU since before I was born and Ive always been fascinated by her line of work, especially these last years since Ive become a programmer myself. Ive been asked to interview her plenty of times, and finally decided to do so.
The banking programming world is a completely different world than what most of us are used to, and for the next couple of hours after posting this Ill accept whatever questions you may have for her, but keep in mind that some questions may not be answered due to security concerns. Ill be posting this to HN & Reddit and will answer questions over there as well.I wont write this as a Q&A, but more like Im telling you her story.UPDATE: Ive added Q&A of some FAQ to the bottom of the post!
The year she started internal training at Nordea, which back then was known as Nordbanken (The North Bank) but changed name to Nordea in 2001. During the training she had to take various tests, most notably an IQ-test to see if she had the intelligence to work within this field. Other tests includes a psychological checkup to make sure she had the psyche to handle that line of work and a multitasking test which she failed horribly where she got the score 22/100. She did very good on the other tests and among the 16 available positions, she managed to get one.
This position is the most important one in the bank, at least from a technical standpoint. If, lets say, my mother and everyone on her team would quit their job, the bank would go under within a matter of weeks if theyre lucky. They have a rotation of people on her team being available 24/7. I remember when I was younger and she had to take a taxi to work in the middle of the night on a Sunday to fix a dead-lock problem.
is not a fancy programming language like your functional Haskell or concurrent Golang its an imperative, procedural language and since 2002, object-oriented. Theres nothing wrong with the language itself, the problem is that barely anyone knows itat least not in the context of mainframe programming. My mother is the next youngest person on her team, and shes born 1964, and the youngest person being 2 years younger. Since almost all of the largest banks in the world runs on IBM Mainframe with COBOL as the primary programming language, this is a global issue. The smaller banks however are better off which usually runs something like Java without mainframes.
My mother used to ask me if I wanted to learn, but since working with the more fancier stuff like Postgres, Redis, Node, Crystal, PHP, among others, Ive always answered Never, ever!. I am still very interested in what she do, but these types of systems gives off the probably worst enterprisy-feeling that you can imagine, which Id like to avoid.I can only imagine the fat paycheck a 20-year old mainframe programmer would get though, because your age in this case would be invaluable.
Their primary database is called IMS, which is an hierarchical database built by IBM for the Apollo program. Internally they call it DL/1 which is short for Database Language One. They are trying to migrate to DB2 which is a relational database that speaks normal SQL, but considering the sheer volume of data that Nordea is storing, this is a task that is going to take years. Its not as simple as just moving the data from IMS into DB2, they also have to update their modules to load & save data from DB2 instead of IMS and they have thousands of modules, many of which were developed by programmers that have either passed away or have retired.
Each transaction is stored in DB2. Theyre avoiding writing to IMS as much as possible, only reading data from it until their newly purchased system is fully integrated and they can start storing data there.
IMS is extremely old and very slow (for some tasks).Searching for data can take hours. Hah, and here we are, arguing that MySQL delivers 2ms better query speed than Postgres. Thats a bit ironic.
They also use a flat-file structure for a lot of tasks, which themselves has different flavors that IMS supports. One such example is GSAM which has forced my mother to take a taxi into work many times due to modules working on the same GSAM-file at the same time, producing deadlocks.
Lets talk about the size of their databases for a moment. DB2 is only storing data about transactions, and each transaction is different in size depending on what type of account the transaction goes to/from. Private accounts like my personal bank account is orders of magnitude more simple than bank accounts used by business. Each transaction is somewhere between 2KB to 500 bytes, so lets say the average is 1KB.At the moment, their DB2 database holds 11 billion transactions, and the law requires them to save each transaction for 10 years, having settled for 11 years. At the moment, transactions are only 7 years old and the amount of transactions is estimated to grow roughly 58% each year, until they hit the 11 year mark where they can nuke transactions that are older than 11 years.
So at the moment, the DB2 database holds about 10TB of data, and that data is only transactions. In 4 years, itll be somewhere around 1314TB of data.IMS holds most other information. That kind of data differs a lot in size, so its hard to estimate how large it is, but I guess its much larger than their DB2 database.
They also store some data on tapes, where their programs tell a machine to load/save data from a specific tape, which is pretty cool!
Everyone has some form of IDE or text editor, right? Well, yes they do. The IDE is called ISPF which is like an entire OS. ISPF can be extended, and the part we would call an IDE is an extension to ISPF which is called Endevor.
ISPF is directly connected to the mainframe, and theres no such thing as a local development environment here.
ISPF Interface found on theweb
Roughly 80% of their systems are batch jobs. These are jobs that runs at a certain time or interval, doing some processing on their data or sends data to other banks/agencies/etc. For example when I buy a can of Coke, the money is withdrawn from my account balance, however the money is not actually transferred anywhere until one of their batch jobs does so. These jobs are usually executed during the night, which is usually why it takes a day before transactions between banks are completed. Transactions to the same bank are usually instant because it executes immediately.
Batch jobs are working with datasets on a gigabyte-, sometimes terrabyte-level and can in some cases take hours to complete. Id love to see how IBMs mainframes all of a sudden starts working at full capacity the second the clock turns 00:00 in their datacenters, would be pretty cool!
Banks that run on mainframes have quite a lot of issues that they need to take care of, but unfortunately time is scarce.Programmers are getting older, not many wants to learn and the time before a new employee can stand on their own feet is 23 years, and even then there are a lot of gray areas for them.There are programs that are decades old that nobody even knows what they do and the person who wrote it is long gone.
Many banks are trying to purchase shiny new systems to replace what they currently have, while almost all of their programmers are saying this is not going to work, you cannot replace this monster that we have here.The reason their systems has become so huge is partially because they made the mistake early on to tailor their own system towards other systems. A great example here is how we build RESTful services, where programs that wants to get data from us have to tailor their system to match ours. This is not the case with some banks, instead theyre sending data to other systems (such as the tax agency) which may at any point update their systems which in turn breaks the current system the bank are using to send the data.
Banking systems are also extremely advanced. A personal bank account differs a lot from a business bank account, and there are at least 50 different types of bank accounts for each of them. And in Nordeas case, they also have the Swedish government accounts, which are different from both personal and business accounts. I think they have the Finnish government accounts and maybe a portion of Denmarks as well, which differs too.
Its going to be very interesting to hear what Nordea and other banks will do these upcoming years, and what new tech theyll be implemented on.I may have forgot to ask something, so if you have any questions go ahead and ask, Ill be forwarding them to her and write down her response for the next couple of hours.
Why did you choose to work with IBM Mainframe COBOL programming?- Ive always wanted to work with computers, but I wasnt aiming for this specific job. I saw an ad in the papers and applied and got the job. I had a bit of computer background before I applied though.
What is the worst thing that youve seen on a day?- My co-worker forgot to add a dot to the end of a statement for a module in the most critical part of our system which we call The Cash Register. It is the part that handles all the money. The result was that the entire bank was down for 16 hours straight due to the module continiuing executing statements when it actually shouldve stopped after that statement. It basically overloaded our systems, a self-DOS of sorts.
What do you think will happen in the future for banks that run on the same infrastructure as Nordea?- Most banks has acknowledged that we need to replace the older mainframes with something more modern. Nordea has bought a new system which it has promised to replace the current one within 4 years, but Nordea takes care of multiple countries and a more reasonable number is 4 years per country, so 16 years in total.Banks and finance that havent acknowledged this is going to have a very rough time ahead of them. I still think well live on IBM infrastructure though.
What were the challenges youve faced as a female programmer that started in the 90's?- No problem at all. Ive got a couple of female co-workers, but most are men. It doesnt bother me that much.
Youve been working for the same entity and possibly the same code base for more than 20 years. Does it ever gets old?- Yeah it definitely gets tiring after a while, like most other lines of work becomes. But during my time here Ive build many completely new systems for various things in finance, and that has always been fun. Unfortunately from now on we will never build anything new, only maintain what we currently have and wait for the new system weve purchased to replace the current one.
How scary is it to write code for a bank?- It is very scary, especially when we push changes to production which happens on Sundays. Whenever we push new changes to production, huge parts of the system has to be taken offline. One of those parts is the entire IMS. During that period anything can happen, but we have a very robust test environment so it usually goes through smoothly.
Have you caused any huge mistakes for the bank?- Definitely have, quite a large mistake I did back in 1997, when my youngest son (thats me, the writer) just started pre-school and my parental leave was over. We have a system for saving money for retirement. Those type of bank accounts wasnt locked back then, and the law says you cannot withdraw money from that account before youre 55 years old. Since the accounts wasnt locked, it was possible to withdraw money if you had the bank account number, so the solution was simple; not give the customer their bank account number.I managed to screw up by modifying a module that added their bank account number to the mail that was sent out to customers. So what happened was that customers started withdrawing money (which was not taxed yet!) from their retirement savings account before they were allowed to.This triggered a huge inspection, the Swedish government stepped in, the financial inspection and the media were all over it. That was me.
Whats your working environment like?- Weve recently moved to a more hip location. We used to have personal desks, but now we have this pick whatever spot is avaiable open area. I dislike it a lot.