Welcome to part #5 of our amazingly blogpost and webinar series! In our series we help you analyze all your Domino applications.
The title of the webinar for this blogpost was “Don’t end up in a ditch because you weren’t aware of roadblocks in your source code”.
This time we are sharing Why, what and how one should analyze in your Domino applications.
A foreword for non-developers
If you are not a developer: The following introduction may help you better understand the rest of this post.
Every database in your environment contains design elements and code. This is to display all therein documents. Be it for creating, editing or reading them.
Views display your documents in list format. These may be sorted and/or categorized. For example, by date, username or whatever fits the respective application and view.
Besides views, two more design elements are worth highlighting. These support you to create, edit or read documents in your databases:
- Actions in views (usually available via buttons or menu options at the top of views)
- and Forms, as well as fields on forms
A database can consist of many more types of design elements. Each of them can use one or many of the following programming languages:
@Formulas, LotusScript, JavaScript and Java.
That code again can have a variety of interfaces and dependencies:
- other Notes/Domino databases,
- other applications, such as Microsoft Excel or SAP,
- the file system – be it on clients or servers –,
- the operating system,
- and many, many other dependencies.
An application may comprise one or many databases and be available on one or many servers. Replication keeps the same databases in sync across servers. This results in great performance, load balancing and high availability across geographies/networks.
When looking at optimizing, modernizing or migrating an application, you must know:
- how complex an application is (think types and number of design elements, as well as lines of code),
- what all the code does,
- and whether the code will be easy to work with in your respective project.
So, let’s start with Why
Most developers did not develop all applications by themselves. Even if they did (chapeau!), it most likely didn’t happen “all yesterday”, but over a couple of years. Do you still remember the code in all your applications well enough?
Also, some companies don’t even have a Domino developer anymore. Let alone all who helped build today’s application landscape throughout the years.
Next, knowing the source code of an application is great for running/maintaining it on Domino as is.
Finally, you want to find all the roadblocks, as well as helpful code, if not in general the good, the bad and the ugly.
If you want to optimize, modernize or migrate your applications:
All in all, wouldn’t it be great if you could save yourself valuable time, frustration and pitfalls? Knowing the stakeholders of your applications, becomes far more powerful when combined with analyzing the design and code of your applications.
Once you know your stakeholders, you’ll also know:
- Who will be impacted by any changes you make to an application?
- How will they be affected?
- Who should you consult before you start making changes?
- Who is receiving the benefit of the work you are doing?
Which brings us to the What
Before we dive into the “What of your code should you analyze”, please remember:
There are many other important datapoints to consider before analyzing code:
- Which are your most used and least complex applications?
These pay off fast. And you should not spend time with applications nobody uses! For more details, also see here and here. - Which end users need local replicas for performance or offline usage reasons?
- Which applications do your VIPs or your most important profit centers use?
These are just a couple of examples to keep in mind – more can be found here (see slides 17-21).
Now what code should you analyze, and what should you look for therein?
All of it. Period. In *all* your applications. And for each of them, that means
- All code: @Formulas, Java, JavaScript and LotusScript
- All design elements (think “code containers”). These can be forms, subforms, views, columns, actions, agents, buttons, script libraries etc.
- The following design elements often deserve special attention:
XPages, Java Classes, Applets, Jar files, Web Services, Composite Application features, and similar
Proper analysis of both application design elements and code answers two essential questions:
- Where does all your code live? How much is where? And what type of code is it?
- What does that code do?
The following two examples show the value of analyzing both design and code:
a) The more forms, fields and code an application has, the more time consuming it will be to modernize or migrate it
b) An application using Java Code is not an ideal candidate to migrate to SharePoint. Yes, this partly depends on what the corresponding code does. Yet, it helps you to rank your applications and prevent pitfalls.
WHAT you should look for in your code
Searching in code can be anything from fun to frustrating.
After exporting the design of your apps to DXL (Domino XML Language), here are three tips to get you started:
- Fire up notepad or similar and explore the result. To get a first feel, try and find some of your code by searching for parts of it. Also, search for usernames, servernames and similar
- Try LotusScript Manager or Source Sniffer from OpenNTF
(do *not* use Lotus Analyzer! It has hidden design and phones home) - Pro-Challenge: Slice the code from the DXL into documents in a Notes database. This facilitates categorization, searching and post-processing (to count design elements, for example)
If you tried any of the above, you may have noticed a couple of shortcomings on the way:
a) Your searches also match comments/remarks in your code
b) Combined searches like @Db(Column OR Lookup) call for tackling above pro-challenge first
(=slicing the code into Notes/Domino documents or a SQL database). Separate searches lead to duplicate results. This in turn works very much against your aim to review as few code as possible.
c) Your searches also match code you were not looking for. For example:
- Searching for “Open” finds NotesDatabase[Object].Open and NotesStream[Object].Open
- Searching for “@DbLookup” includes lookups into the same database where the code lives. Yet, you might only want lookups to other/external databases. Or, your result also includes non-Notes lookups, like ODBC. Yet, you might only seek Notes lookups.
So, before we continue our search, we must optimize the code
For good search results we must remove any and all comments from all code. Yes, comments are good to have in your code to better understand it again later. But they skew our search results.
The following pictures illustrate how a developer can comment in LotusScript and @Formulas:
In @Formulas comments begin with a REM, followed by any amount of whitespace (=blanks or tabs). Next comes the actual comment surrounded by either double quotes or curly brackets.
We’ve already prepared the above for you to test for FREE
This entire article helps you understand and perform your own, independent application analytics. Yet, you may want to save yourself valuable time and get moving fast. All you have to do is register for our ready-to-play iDNA Applications sandbox. As soon as you’ve registered, all you need to do is login and navigate to our ready-made instant code search. Granted, it’s not showing your very own applications, but gives you a good idea of how code search should work.
For testing, use text search using “florian”, “vogler”, “server”, “/acme”, “/O=acme”, and “workflow”. Also, try a regular expression search like “(?iw)@db(lookup|column)”.
Now let’s really dive into WHAT to look for *in* all the (design and) code of your applications:
If you want to optimize
- Search for GetNthDocument – it’s slower than GetFirst/GetNextDocument
- Search for hardcoded servernames, usernames, database filenames, IP-Addresses, email addresses, replica ids, etc.
- Search for old code (@V2If, @V3Username, @V4UserAccess, @UserPrivileges, @IfError)
- Independent of code: Find your most used/popular databases and give them a little love. Make them prettier and more modern!
If you want to modernize
- Check out whether an application heavily relies on NotesUI* classes. That doesn’t work in web browsers and needs some rework
Check for code that’s not supported in browsers. Hint: search Designer help for “You cannot use this function in Web applications.” - Is there already a lot of code suggesting browser support?
i.e. @WebDbName, @BrowserInfo, @ClientType, Domino @DbCommands, … - Does your application/code rely on printing? That can be challenging from a browser
- Analyze your code for whether it works on HCL Nomad
- Does the application depend on XPages, Java or ODBC? This does not work on HCL Nomad.
- Does the application use C-API calls? If yes, does the code also work on iOS and Android?
- Independent of code: find your most used/popular databases and give them a little love. Make them prettier and more modern!
If you want to migrate
- How many forms, fields, views etc. does an application have? Don’t pick the most complex first.
- Does your app depend on code or design elements that do not work well in your respective target platform?
For example: Java <> SharePoint, too many Folders <> SharePoint. C-API. Private or public Addressbook, Mail (Send and) Encrypt. UseLSX, ODBC, DB2, DOS/cmd, OLE, files, directories, MIME, document links, etc. - Does an application depend on other databases?
Think @DbLookups that connect to other databases (not using e.g. ““:““ as server:filename, for example). Same goes for New NotesDatabase, GetDatabase, .Open, .OpenWithFailover, .OpenIfModified, .OpenByReplicaID, [FileOpenDatabase], [Compose] etc. - Independent of code and migration: give one of your left-over databases a little love, make it prettier and more modern.
Finally let’s look at HOW to best search your code
In part, we’ve already covered why searching for code with just (sub)strings isn’t going to get you all too far. It matches too much or too little in too many cases. We also looked at why removing all comments is necessary before searching your code.
Also, the following is very, very, very important when searching for code:
Do NOT search for code thinking of how you would have developed it. Think broader and you will be quite surprised to learn how other people code.
So, what are better approaches for searching for code than just substring matches?
A fulltext-index supporting wildcards like „*“ (asterisk) gets you somewhat further in the game. E.g., when searching for „@dbcolumn* OR @dblookup*“ – but it lacks support for precise negation. Accurately negating code parts is vital to e.g., only find @DbLookups that do not point to the same database.
The following @DbLookup looks up data from the same database that the code is executed in. It does so through the ““:““ as the second parameter:
The next @DbLookup looks up data from the „local“ addressbook (client or server, depending on where it runs):
Searching for any @DbLookup that looks up data from “names.nsf“ is pretty easy. Even with just wildcards: @dblookup(:“names.nsf“). That’s until you come across code like this:
Now we suddenly need to search for code across multiple lines – yes, we’ve seen code like that.
Where it gets even worse is when variables, let alone @functions, come into play as parameter:
The above code looks up data from the same database. The variable ht_filename is the result of @Subset(@DbName;-1). This again results in the filename of the database in which the code runs.
Similarly, the following code example looks up data from the local addressbook:
The best solution to searching code would be if one could parse the code. This would allow us to resolve the value of ht_filename in above examples. However, we have not found a smart solution for this.
What we did find a smart solution for is searching for code with precise negation:
We use regular expressions.
The following regular expression is a huge step forward. It allows us to look for any @DbLookup that does not lookup data from the same database where it runs from:
@dblookup([^;];(?!””(:””)?).*| @dblookup( | The open bracket needs to be escaped in regular expressions, hence the \( |
| [^;]*; | followed by „anything but a semicolon“ until the next semicolon. Searching for „.*;“ would be wrong since that expression would be greedy. It would search until the last semicolon. |
| (?!””(:””)?) | Next negating ““, followed by another optional :““ – the second parameter can either be ““ or ““:““ |
| .* | until end of line |
This example still requires some fine tuning to
- also support any amount of whitespace pretty much everywhere,
- and cater for situations where @dbname or @subset(@dbname;1):@subset(@dbname;-1) use the same database
- and only find those @DbLookups for which the Class is ““ or „Notes“ (case insensitive)
As bonus points, you may also want to find @dblookups used in LotusScript Evaluates. Oftentimes, quotes need to be escaped then, too.
The regular expression which matches all the above requirements (except for code parsing) is … possibly the ugliest thing you will get to see today:
I could go on and on for hours
If you made it through the really creepy stuff above: my hat is off to you! You are a superhero developer and regular expression survivor.
The good news for all readers is: We at panagenda have already done a good deal of the heavy lifting. And, we love sharing.
If you take a look at our iDNA Applications sandbox, you will find more than 70 ready-made regular expressions. These search for over 300 different code findings. You can use all those patterns in your very own code search solution for FREE.
And just in case you have a smarter idea than using regular expressions or come up with some new or improved regular expressions: Please let us know and we will in turn share with the community.
In case you don’t have the time to build your very own code search solution:
There are a number of 3rd party solutions out there that can help. Some of them are available on OpenNTF. Some of them are commercial solutions like our very own panagenda iDNA Applications.
Summing it up
Properly analyzing your applications serves three primary goals:
- Saving (a lot of) time, frustration and money
- Focusing on the right things and being in the know
- Enabling the successful transformation of your Domino landscape
Optimization, modernization or migration
- Whatever you do don’t forget to give at least one of your Domino applications some love. It will pay you back. Many ti
Next in our series
Progress reports. They’re a part of every project and they’re not fun to produce. It’s not just your progress you have to share. You must also supply the project teams with the data they need to do their job. It’s tough to coordinate and it never ends. But progress reports can be your friend when you’re reporting success!
Migration and modernization projects are hugely expensive and very high-profile. From the very start, a lot of eyes will be on you. Expectations will be high. How can you keep your project from failing? Sometimes you can’t. Some projects are doomed before they even start. You should know before you go.
From here on out it will be a continuous widening of the circle of applications we are working on. Each step will be more cost intensive, and prudent planning will yield ever greater benefits.
About this series:
Many companies around the world have been committed to HCL Notes/Domino* for years. They know the many benefits that come from that relationship. Additionally, Notes/Domino lies at the center of their processes and how they work. Despite all this, IT decisions makers around the world are starting to envision a future where Notes/Domino may play a reduced role or no role at all.
*formerly IBM Notes/Domino
