This is to discuss with other PBSPro devs on how we should go about solving race conditions which occur because of the way the scheduler currently queries PBS data from the server. Currently, the scheduler makes multiple queries to the server to finally piece together all of the information that it needs from it, which means that the scheduler can potentially have inconsistent information if the data changes in between the different queries.
So, I wanted to discuss a solution to this. The fundamental issue is multiple queries, so it definitely needs to be a single query.
A couple of possible solutions which came out of a short discussion with Bhroam, Bill and Subhasis:
Store things to a database which can be queried for all of the data that the scheduler needs in a single query, with transaction atomicity. Would be great if one of the standard DBMS softwares can do the job, or we’d have to come up with our own data service to cater to any specific PBSPro needs. Possible drawback: this might require quite a bit of refactoring and take considerable effort and time.
Create a new IFL call which can return all PBS data. Possible drawback: might make things tricky if we multi-thread the server in future.
What do you guys think? Any other ideas? Any comments on the two solutions mentioned above?
Having some sort of an API that returns a consistent state of the system is a good solution. However, an even more flexible solution could be being prepared for a bit of state inconsistency, and reconciling things as we go.
There are many real world systems that use this approach. One example of not a true OLTP application are hotel bookings. Sometimes we load data to edit and by the time you go to update, the api first checks that the data is the same as was last refreshed and only then updates - if it detects things changed, it reconciles. So if we play the rules well, it might be possible to be able to not depend on the state of the world being consistent across our queries from scheduler.
That’s an interesting thought, it reminded me of cache coherency protocols in multi-processor systems. One could visualize each atomically obtainable data set in PBS as a block, which would be marked “dirty” if it gets modified, and the scheduler can do this check: in the time that I queried all the PBS data blocks, did any of them got marked “dirty” by the server? If yes, it would reconcile as you said. But I’m not sure how the scheduler would reconcile things other than querying everything again. Or maybe we can classify what got modified and how, and decide if we need to query everything again, or just one/few things. But that would add complexity and may also introduce race conditions. So, if the “query all pbs state data at once” is not an expensive operation, I still like the database solution better, it’s simple and guaranteed to erase race conditions.
Don’t reinvent the wheel in PBS Pro if we can take advantage of functionality provided by external components like the database. I believe it’s time we reconsider how we utilize PostgreSQL with PBS Pro. Do we need a relational database providing ACID transactions or something more distributed with builtin failover? We do not take advantage of many PostgreSQL capabilities today, and maybe we should. Would we be better off looking at alternatives like Redis, LMDB, or NoSQL?
Lots of factors to consider, scalability being paramount in my opinion.
One of the reasons we haven’t moved to a direct database query in the past (besides having no time) is that the server encodes the data. Some things like eligible time are encoded on the fly and reported. We’d have to move this encoding to the database before we could use it.
I do agree that not reinventing the wheel is very attractive. Databases know how to do this well. Our solution would probably not be as good and maybe take us longer to implement.
I share Mike’s concern about scalability, I think once we start moving the “data services” role from the PBSPro server to a database service, more and more ‘clients’ will start using this service and scalability will become very important. So, whatever solution we think of should be considered for long term use. So, I’d definitely vote for distributed databases, which sort of makes PostgreSQL not a good candidate right? It’s not a DDBMS.
But I’d like us to conclude one part of the discussion before we start discussing databases. Do we all agree that moving things to database is the best solution for the problem of race conditions? or is there any other solution which might be able to solve this particular problem just as well?
A database represents one possible solution to address race conditions. PostgreSQL would help with this, but distributed databases often would not. In terms of performance, a properly configured DBMS backed by a high performance data store (e.g. SSD, RAID5, etc.) would probably be sufficient in nearly all cases. However, it still represents a single point of failure.
“PostgreSQL would help with this, but distributed databases often would not” I think that’s debatable, DDBMS like Cassandra and Dynamo DB can be configured to follow strict consistency rules, which would be bad for performance, but would definitely solve the race condition issues. But maybe some form of eventual consistency model will also work, which is where DDBMS shine. What do you think?
It’s definitely worth an investigation. We could start by constructing a “wish list” of features PBS Pro might take advantage of if they were available. PostgreSQL represents the incumbent candidate, but there are many (pseudo-) databases to consider. Data consistency, overall performance, and fault tolerance would be at the top of my list. It’s my belief that the current client/server model PBS Pro uses today should evolve into something more distributed with multiple servers and schedulers. Using the right data management system could save a lot of work by preventing us from having to reinvent the wheel in many cases.
Thanks Mike, that sounds like a good direction to me.
Alright, with no other inputs on the topic, I think we can conclude the current discussion:
Solving data race conditions should be done via databases. Looking at a more distributed future and myriads of other purposes that a data service could serve, the choice of database to use deserves a dedicated discussion/investigation.