cancel
Showing results for 
Search instead for 
Did you mean: 

Persistence Layer

akent99
Regular Contributor

This post is first in a series talking about planned improvements to the persistence layer in Magento 2. It provides an overview of the scope of work planned. This work is still reasonably early in the research cycle and we are looking forward to community engagement as we proceed.


What? Another ORM?

 

Starting from a traditional database access perspective, Magento 2 business logic has SQL queries directly embedded into the code. One of the areas of investigation is to improve the separation of business logic from persistence logic. This has a number of potential advantages, including exploring alternative database technologies for improved performance. This separation of business logic from database access code is a traditional benefit of ORM technologies.


The persistence API we envision for Magento however is not a simple ORM like Doctrine. Traditional ORMs deal with mapping rows from relational database queries into in-memory objects. Good ones also deal with schema changes. The Magento persistence layer will go beyond such ORMs. The Magento persistence layer understands Magento concepts such as entities. The Magento Entity Attribute Value (EAV) model will be managed by the persistence layer (business logic gets a consistent view of all entity attributes without care of the physical storage representation).

 

Going further, the importance of software patterns such as the Command Query Responsibility Segregation (CQRS) pattern are becoming more widely acknowledged in modern software systems. CQRS separates "commands" (which result in modification to master data) from "queries" (retrieval of data which may have been transformed / aggregated in some way for faster retrieval). This has direct alignment with the Magento EAV master data model (modified by commands) and indexes (generated for supporting more efficient queries). Indexing is also managed by the persistence layer (business logic does not have to care about what indexing approach is used and how to trigger it).

 

So, while the persistence layer in Magento does include traditional ORM functionality, it goes a lot further.


Goals


There are multiple goals for this separation, but a high level the overall intent is to open up the possibility for more experimentation in use of different persistence technologies. Do you want to use Oracle instead of MySQL? Sure. Do you want to try MongoDB or ElasticSearch for your catalog index? Go for it. Do you want to use an in-memory product catalog? Fine. Separating business logic from persistence more cleanly will make such cases easier to do. (Not “easy”, just “easier”!)


More specific examples:

 

    • Improved CQRS alignment - separation of "commands" from "queries", recognizing the data model for each can be different.
    • Allow different indexing strategies to be plugged in without impact to business logic.
    • Allow different persistence technologies to be used for different tables (MySQL, ElasticSearch, etc.).
    • Align service contracts with persistence data structures providing a standardized query (and most likely update) API across all entities, and to improve performance by allowing the same data structure to propagate from the persistence layer all the way up to the presentation tier without data copying.
    • Introduce a higher-level abstraction of the persistence model than the raw database tables – e.g. “Customer” as one entity rather than talking about all the EAV tables behind it.
    • The higher-level abstraction of data model includes separation of “master” from “query” data model views.
    • Database schema upgrades are automated – schemas are declarative and schema differencing is used to automatically generate schema upgrade commands.'
    • Flat tables can be used for master data attributes for improved performance. Knowledge of the richer Magento entity data model is built into the persistence layer.
    • Allow higher performance bulk import of data, either through the persistence layer or by providing sufficient metadata for such tools to use.

Conclusions

 

The persistence layer work is a significant fundamental platform work item. It is not going to be rushed. For myself, I am excited by the new opportunities it will open for Magento and the community to push the boundaries of performance by allowing special purpose technologies (like in-memory databases) without change to business logic. It is this standardization and decoupling that excites me the most about this work stream.

       

Future posts in this series will dive deeper into specific concepts to share plans and gain community feedback as plans proceed. If you are interested in this area and want to be involved, please keep your eyes open for future posts in this blog.

 

8 Comments
Navarr
M2 Certified

From a developer standpoint, this is very interesting.

 

One of the most important aspects of this I'd like to read, however, is: How should we write code today to most easily refactor to the coming system.

 

Obviously a lot of it is still under consideration and may be changed, but it'd be nice to know what we should ensure we keep separate so that it can be more easily refactored in the future - even if it's just "ensure your business logic happens in a repository class and don't rely on resource models."

akent99
Regular Contributor

Easiest suggestion is to make it easy to separate the PHP class holding data from business logic, and try to keep SQL queries isolated well. The new persistence layer will have a separate data structure for holding entities in memory (not the active record model where the attributes and business logic are in the one class). At present if you are messing with products and multiple scopes you have to do more sophisticated queries your self - we are still going to allow query building, but it will be on top of the entity structure (we will map to all the join queries required under the covers for you).

sivakumar34
M1 Certified

@akent99Really nice to see the future Magento ecommerce..sorry Magento Framework...sorry Magento Software..sorry Magento Robotics.

But as a client perspective we need system with minimum bugs,please plan accrodingly.

Later we can think all these.

In github in a week opening issues around 50 ,closing issues around 30.

I want to see Magento 2 Market share above 50%.

 

akent99
Regular Contributor

Thanks. We are investing signficantly in bug fixes as well. Max and team have been working through the GitHub pull requests - about half are now closed over the last few months, so making good progress there. https://twitter.com/maksek_ua/status/846392591990083586. As an open source project, we want to better leverage the power of the ecosystem to contribute both bug reports and fixes. But we have also been working through internal issues as well and upping the quality of 2.x patches.

In addition to that, we need to continue to invest in the platform to make it easier to use, which typically results in fewer bugs in the first place. This database layer work is a part of that effort. Simplify and standardize.

 

akent99
Regular Contributor

Hi @sivakumar34, I am sure I sent a reply, but I cannot find it here so apologies if this is a duplicate. You get to see consistent I am/am not! ;-)

There are several activities going on relevant here:

  • Agreed. We have been throwing a lot of internal resource on bug fixing and will continue to do so. The more recent 2.0 and 2.1 patches demonstrate the increased rate of delivery, and that is not stopping any time soon.
  • Max and team have been burning through the pull request backlog. If you look into the stats you will find they have been doing some amazing work. In two months the backlog has been halved - they are processing pull requests much faster than the arrival rate.
  • Core Magento is an open source project. We are currently focussing on getting pull requests in faster so the community don't have to wait for us to get a bug fix. We hope the community will help more with both submitting issues and fixes for those issues. We felt it more important to enable the community to get pull requests in and accepted more efficiently than trying to fix everything ourself. Max and team are spending more time training external groups on how to do a pull request done well to help us streamline faster.
  • So the ultimate strategy is not just for Magento to invest more into bug fixes, but make it faster for the community to help fix the issues as well.
  • Projects like the persistence layer project do actually help with bugs. Simplifying and standardizing helps reduce the number of bugs in the first place.

So we do have a multi-fasciteded solution to improve upon where we are at. We are absolutely dedicated to quality of the platform and are moving forwards on multiple fronts to achieve this.

Thanks for the feedback!

sivakumar34
M1 Certified

@akent99  i understood now.Really sorry for my comments.I will try to contribute as much as i can as a community member.

 

oblil
New Contributor

Show invalid scope or scope id message.

I'm running Magento 2.2.2 on a shared server and just did a new install. I am using the standard Luma template and when I attempt to change the site logo Content>Configuration>Edit>Header>Upload Image, no matter what format or size of the image, I end up with an error stating Invalid scope or scope id and cannot save the image.

 

I'm new to Magento, coming over from Wrodpress.

Any ideas where to go from here?

mageklein
Senior Member

Hi, any news with the data persistence project?

 

In the code (2.2.x) it's explained that the entity manager should'nt be used directly in our code, because it will be refactored. I hope that the new entity manager will have the same features (operators handlers, entitites definitions, metadata...). These features are awesome because opeartions are splitted in singles class and are more readable.

 

The current entity manager has a generic approach thanks to the type resolver. It will be great if the entity manager could be used by the abstract resource model to persist the entities (today we need to override the resource model and configure it via $this->_init(...)).

 

Also, what's the purpose of the RepositoryFactory? As repositories does not implement an interface (eg some common methods might be defined, as: get, getList, save, delete), we shouldn't be able to retrieve repositories because the repository result in unknown type. So the repository factory seems useless, even it could be great for generic purposes.

 

Last question, with the current entity manager philosophy, what the purpose matter to extends the AbstarctModel in our models? Its purpose was to allows CRUD actions to the object, but these methods are now deprecated.