JADE  Environment Development Ideas

Source control overhaul

For a few years now there have been discussions on how Jade could be improved by including a new and modern take on source control, utilising industry accepted tools or ideas.

A vote here is firstly then to confirm you still feel this is important. Furthermore feel free to add comments, as it's a wide ranging topic so it's also about agreeing as a community what minimum viable functionality would be needed out of the gate, in order for you or your team to start using it.

Some thoughts below:

Recording changes, being able to:
- capture all changes made to an environment by a user, or set of users working on the same piece of work.
- take a snapshot of some changes (commit), recording along with it the user and a description
- selectively state which changes are in a given snapshot, along with the capability to undo or repair the last.
- select any existing snapshot, as the starting point for any new set of changes.
- understand what change set (branch?) you're currently on, and whether any changes have been made which have yet to be put into a snapshot.

Viewing/Editing snapshots, being able to:
- view all changes in a readable and consistent form (so without the environmental info & ddbs that have plagued existing extracts)
- merge any two snapshots easily. Where an edit is obvious, these would be automatically done by the system - or something external to it - with allowance for manual edits too
- view the history of an entity (rather than a change set)

Moving between snapshots, being able to:
- automatically apply code changes, initiate data transitions and run scripts where necessary to move to any existing snapshot defined in the system. Ideally complex changes such as renames and moves would be supported.
- where failures occur, provide a means where we can retry, or at the least, not be left stuck.
- provide an extract (scm/ddb/jcf equivalent) so another (eg test) environment could transition in the same manner.

Other, being able to
- allow for rule enforcement, so limiting who can load into an environment, or ensure that its not the same person who did the changes
- drive all commands via api/command-line, allowing for tasks to be driven by in-house or 3rd party tools.
- integrate with cloud tools, so as a community we could code share on github or the like (push/pull etc)
- extract the complete development history, so this could be imported into another environment (may get for free with cloud integration)

The ultimate end goal would be where the multi-user developer experience is maintained, so full support for two or more devs to work on independent changes in the same environment at the same time with as little friction as possible. It's likely that this would be non-trivial, but if implemented it would be a great step forward for the source control functionality and the product in general.

  • John Beaufoy
  • May 1 2019
  • Planned
HistoricalComments
Murray Symon 22/11/2017 10:25:07 AM
Git integration is a vital requirement for our current toolset and processes. Jade is now the only thing we don't have in Git.

Delta360 21/10/2017 3:52:03 AM
Absolutely agree. Jade source control is inadequate for multi-developer environments.

Troy Fowles 9/10/2017 10:07:48 AM
Helpful point of reference to see how others are handling databases in source control systems?
https://www.red-gate.com/products/sql-development/sql-source-control/

https://www.troyhunt.com/rocking-your-sql-source-control-world/

John Beaufoy 29/09/2017 1:02:30 PM
@Ben ? thanks and totally agree - tried my best to be technology agnostic :-). My intent is not to define what is needed to be built into the Jade product, but rather define what we need when using the Jade product. The smartest changes to the product may be a combo of new APIs, UI extensions & cleaner extract format, so the wider community (Jade Plant certainly not excluded) can build out the remainder. This would then allow TFS integration or anything else.

Ben Butterfield 29/09/2017 1:48:55 AM
Very comprehensive post John. For me the biggest benefit would be to be able to use available source control solutions which have already solved many of the concurrent development problems we have. You mention git which is the clear industry favourite at the moment and I know you and your team have made progress in prototyping this. From a practical point of view we may also like TFS compatibility as it is already established here for our .net code, I'm hoping any solution would be available for both (and others if other people have established systems).
  • Attach files
  • Gaurav Chaudhari commented
    07 May 21:06

    To extend on some specifics regarding what functionality I would like to see for effective source control integration:

    • For integration with version control such as Git, changes made by each developer needs to be recorded at the most granular level as possible. This includes all things such as compiling changes on a method, structural changes such as adding/deleting/renaming entities on any level (Class/Schema/Interface/RelationalView)
    • This level of granularity will be largely different from current functionality, with schema extracts. As such, a lot of associative information such as class constants, global constants, locale information, and translatable strings will be lost and needs to be handled separately. Possible ways this can be done is by maintaining a backend database that constantly holds the structural integrity of the current source code, or having dedicated files that store this information.
    • Currently, we have our primary development environment setup with Data. When structural changes are made such as deleting a reference, the data for that reference for all instances will be deleted/purged. Scenarios like this we cannot selectively repair by undoing changes (adding back the reference). This is a concern when developers are working on different features residing on separate branches. If in one branch the "reference" is not available or deleted, the persistent data for that Jade environment would be deleted. However, when we switch over to the other branch where the reference is present, the user/developer expects the instances to still contain the data associated with that "reference". Simply put, maintenance of persistent data might need to be given special consideration while making changes granular or atomic in nature.
    • A reliable log of all changes/commits that are made. However, rather than containing snapshots of the current schema structure or the full method source, these changes should have a similar approach to version control such as Git, so as to record the source code lines added/deleted relative to a base/master branch. Similarly, it would need to record additions and deletion of attributes/references relative to a base snapshot, rather than making a whole new snapshot of the current schema. Furthermore, they would, of course, maintain some basic info such as the user/developer who made these changes.
    • Being able to encapsulate these granular changes into commits, which would have the bare minimum information such as time, date, developer, GUID and a small description. These commits would then be easily able to be reverted, transferred to a new branch, merged or split into commits, existing features of established version control systems. 
    • Git/SVN/TFS integration - The splitting up of these changes and features needs to be fully compatible with these existing systems. The specific format in which how hierarchy is stored, data-types and source content are maintained is still up in the air and open for discussion. But once fully integrated with a proper structure, these granular changes should be seamless with a versioning system. If a corresponding database or file is updated with every granular change, this would mean that every time a method is compiled, a basic command/structure change has been done (such as adding an attribute, defining a constant, deleting a subclass) : Immediately a file needs to be generated to indicate the delta/change that happened. There should effectively be no reason a user needs to extract or load data for simply committing changes. When pushing or finalizing changes, more complex operations may be left such as re-organizing data structures and persistent data formally for it.
    • Code Diffs! The more atomic changes are recorded, the easier it becomes to generate these. Functionality should be extended to diff between any two unique commits/GUIDs such that we can retrieve all the operations that occurred in-between, which includes not just source changes, but also structural changes, perhaps a data type change for a certain RPS field in the RelationalView

    Of course, since maintaining the Jade environment is not the same as a general IDE, we need to consider all aspects such as variation in code, change in data structure, change in SQL mappings through RPS, Form changes and possibly further such as jade environment variables and information associated with 'singleton' like entities like system, app, global, node and process whenever applicable.

    Having said all that, these are still simply starting thoughts to integrate Jade with effective version control. Coming from using GIt in the past, I can't help but hope the possibilities and efficiency that can be achieved by implementing this properly!

  • John Beaufoy commented
    09 May 09:26

    @Gaurav with re your 3rd bullet point (Data support)
    Running with several versions of the same classes and objects in the same environment might be quite an undertaking? That would be my only reservation for it being on the list of minimum viable requirements.

    Furthermore there are some existing strategies and some new ones that would come from having the above basics in place that might reduce the need for this:
    - utilising generated data rather than copies of live dbs.
    - consideration for using more than one development environment
    - versioning of ‘breaking’ db changes (eg. property removes, class moves etc), that all other dev branches are then pegged against.
    - use of migration scripts (both forward and back)
    - restoring from backup and ‘pull’ your code changes back in again

  • Admin
    Ashley Bass commented
    12 May 21:37

    We are planning to run a Source Management workshop in conjunction with our TOI in June... hope you can make it and share your ideas and workflows!  Either way thanks for this useful feedback.  

  • Kevin Saul commented
    15 May 02:40

    These latest comments highlight the need to maintain a clear line of separation between the IDE and source control.

    At the initial 'source control summit' meeting some of us attended (circa 2016?), JADE discussed how they were apprehensive of plugging the IDE into any particular source control system due to our various needs & preferences.  By the end of this, we left with an understanding that JADE would provide platform improvements to support those implementing source control using any of the various options available (i.e. friendly extract format without user or environmental details).  However, further consultation didn't transpire and it would seem the approach currently favoured by JADE is to plug Git into the IDE directly.

    I'm not saying JADE shouldn't implement a source control plug-in (they need a system that works for them too!!), but the approach taken needs to be a strategic one as some of us already have well established processes using other systems (potentially built in-house).  While any source control solution provided by JADE will be useful to those who don't already have such processes in place, those who do are unlikely to migrate unless it can replace or integrate with issue tracking, change management & automated release/deployment systems already in use.

    If JADE were to prioritise implementing IDE extensions, this would not only become the vehicle for JADE & the wider community to implement multiple source control plug-ins, there'd be added value due to the platform improvements needed along with the general ability to implement extensions for various purposes (integrating with issue tracking & change management being one).

  • Kevin Saul commented
    15 May 03:33

    Link to the JADE IDE Extensions idea may be handy :)

  • David Wilkinson commented
    27 Aug 20:38

    Sounds really good - merging of snapshots would be valuable and a huge timesaver for our team.