FLEXlm latest information by CrackZ
Introduction
This document discusses the reversing of 3 FLEXlm protected programs. This essay is a classic example of where 1 vendors choosing not to buy into Macrovisions 'enhanced security' aided in the discovery of a very simple (and virtually generic) technique to bypass vendors that had. Throughout this essay I use * Tip * sections to help you through.
Bitplane Imaris v4.0.3 (FLEXlm v9.0.0), v3.2 could be examined for background information.
GerbTool v13.0 (FLEXlm v8.2a).
SDS 2 v6.318 (FLEXlm v8.1a).
* Tip * - Use lmtools.exe, Utilities tab (sometimes shipped with your target application or inside the FLEXlm SDK to identify the version number of the FLEXlm library used). Files with FLEXlm can also be searched with something like UltraEdit for the strings 'lmgr.lib' or 'liblmgr.a'.
I started out on this project with several pieces of background information.
Bitplane Imaris - I knew the vendor keys, seeds and feature names all from v3.1/2 (which used FLEXlm v7.0e) and none of the newer FLEXlm features, I recommend downloading the files from the Bitplane WWW archive here.
GerbTool v13.0 - In my possession was a patch which apparently deprotected the program, initial analysis of the changes confirmed to me that the FLEXlm routines had not been patched, instead higher level checks had been forced.
SDS 2 v6.318 - In my possession was a 'warez released' crack and a (presumed invalid) FLEXlm license. I knew from an end-user that SDS had switched to using a new style FLEXlm SIGN= license several versions before.
I started out with Bitplane Imaris, being that I knew the vendor keys, seeds and license file format, hoping this would make it easier to identify everything that was going on. I constructed a dummy license like below (based on the format I knew in the previous version, also assuming the vendor name 'bitplane' had not changed, since a daemon of the same name ships still with the program this seems like a good assumption) :
In the past with Imaris, a fake license had always given a friendly message box complete with the feature name the program desired and the FLEXlm error code, no longer is this the case, instead we are just booted to demo mode. We need to find _lc_checkout().
Finding _lc_checkout()
Finding _lc_checkout() (when you know how) is actually very simple, however I hacked about inside my targets inside SoftICE instead of realising the one place it had to be and would be identified by IDA, lmgr.lib, there we go good developer link in our lmgr.lib and you too can tell everyone where your checkout routine is as well. IDA does a really fantastic job of decompiling lmgr.lib (be sure to select lm_ckout.obj for the object since there are more than 60 packed inside, you can examine the others later if you are so inclined). Inside you find the string reference 'lm_ckout.c', this cross references pretty much exactly (right down to the offsets of each part of the routine) with the checkout code found in all 3 of my targets. From here its a short step to identifying the real call doing the work _l_checkout().
* Tip * Note you can generate signatures from lmgr.lib using IDA's FLAIR tools, however the signatures are not without faults and often miss key functions, _l_sg() will almost certainly be missed due to its similarity to the redundant _l_svk() which GlobeTrotter leave in for obfuscation. I recommend using my 'lmgr.lib in IDA' approach.
From here we get the unsurprising version number (4.0) and the feature name ImarisBase as well as a job structure. Looking beneath _l_checkout() (inside the disassembly of lmgr.lib) we can quickly trace down an old friend, _l_sg(). Currently _l_checkout() refuses our fake license with the error -8 (the infamous LM_BADCODE). Using Hiew or any other good hex editor we do a search for F8 FF FF FF (-8 in reverse byte order) to see likely places this is set.
As it turns out we end up with 3 locations; in Imaris these are 5F7282, 5F77D7 & 5F8AE5, the first 2 are referenced inside _l_good_lic_key(), the latter inside _l_ckout_ok(), we could patch these but for now lets have a trace through _l_checkout(), the initial order of events is as follows :
FLEXlm aficionados will remember this convention from the old days of _l_sg(). Next up :
_l_valid_version() - check for valid version.
_l_clear_error() - clears the error from _l_ckout_borrow().
_l_zcp().
_l_next_conf_or_marker().
_l_local_verify_conf().
_l_good_lic_key() - sets the -8 error.
Once again, we get some trivial setup code, configuration verification and then _l_good_lic_key() sets our error.
_l_xor_name(bitplane) - l_xorname(job->vendor, &vc);
_l_sg() - l_sg(job, job->vendor, &vc); (did you know that l_sg means "signature vendor_key5" ;-) ).
So our old friend _l_sg() is here, in order to verify whether anything had really changed I decided to apply Nolan Blenders technique of seed recovery.
Before tracing over _l_sg(), be sure to note a copy of the vendor code structure, in my case it was like so, note also that the job structure is probably not filled in :
Plugging these values into calcseed.exe we successfully derive 0x0A192837 (seed 1) & 0xF0E1D2C3 (seed 2), these match the seeds from the previous version. Generating vendor keys using an updated version of lmkg.exe (courtesy of good friend Nolan Blender or using lmv8gen.exe (found somewhere on my FLEXlm page)) we can generate licenses (note that you'll need to insert defines for ENCRYPTION_SEED1 & ENCRYPTION_SEED2 or copy them from an older lm_code.h), I also found in later SDK's LM_SEED's must also be defined, although they won't be used if you set LM_STRENGTH appropriately.
Bitplane will now check this license out successfully, this indicates that it does not use any of the new enhanced FLEXlm features even though all of the routines are not compiled out, you can now freely generate licenses for the remainder of the features, accessible from the very convenient license option.
A Snag
FLEXlm couldn't be this easy?, surely?, lets try the technique above with GerbTool.....As it turns out GerbTool tries to check out 7 features :
gt-fab / gt-dsn / gs-ins / gt-com / gt-vwr / gt-vdb / gt-ce (gt-odb & gt-pad are checked out by ODB2GT.exe & PADS2GT.exe respectively)
I was informed each of these corresponds to an edition type of GerbTool, so only 1 feature should actually be checked out successfully. I'd generated my license for gt-fab and _l_checkout() rejected it with error -8 (LM_BADCODE), the other features failed with -5 (LM_NOFEATURE), no real surprise there, crucially we now know there must therefore be something else beneath _l_checkout() that determines that the license is invalid.
I was pretty confident that the reason the license was rejected was due to the new ECC security, since this has been well implemented (Certicom did their job correctly), there is no hope of finding the real LM_SEED's. A brute force attack would be 2^96, inpracticle to say the least (and rendered further so by the speed of the authentication code). There are now 2 things to do, first we can probably tease from the FLEXlm library the -8 error (0xFFFFFFF8) and find where its set (like Imaris, a hex editor will do for finding those places in GerbTool), as it turns out there are 6 address candidates, simple testing would quickly find the correct one. Or, secondly, we can just trace on from _l_sg() with GerbTool.
Tracing onwards I simply compared the code flow from Imaris with that of GerbTool, both targets next call _l_ckout_crypt() and code flow differs soon after a call to _real_crypt(), modifying execution at this level produces a null pointer error. With Imaris code flow is directed towards a familiar side-by-side comparison of the real license with the one obtained from the license file, this implies that _real_crypt() and routines below must be deciding whether to perform the old style checkout or the new style authentication. Lets look at the piece of code that controls this (from Imaris) :
The value of EDX at 5F9B56 controls which type of license 'check out' will be performed, with Imaris the value was 0x1048C0 and with GerbTool 0x48C0, if we now modify this value in a live debugging session, GerbTool will check out using the old style FLEXlm encryption and we'll be licensed successfully. The next step is to see where the FLEXlm routines set this flag. With a simple bpm we can backtrace our value being set using an 'or ecx, 100000h' instruction inside _l_good_lic_key(), this is referenced back from another flag checked just after _l_sg().
So if we've got a 0 here we perform the old checkout, we could now just settle for patching here but lets backtrace yet again. This results in 2 locations that write 1 to our [ecx+51Ch] flag, inside GerbTool these are 6AF454 (at the end of _l_init()) and 6B219C (at the end of _lc_new_job()).
Flag 1
6AF454 - Set to 1 using a static ADD ECX, 1 instruction at 6AF426, this could be patched to ADD ECX, 0 for a 1 byte change however Imaris has the same reference so there must be a variation in code path. By comparing the flow between Imaris and GerbTool, we discover the following 'switching' code :
This is the real switch we have to backtrace and its set deep inside the _l_buf_36() (routine described below), the only way I reliably found to locate where the static value is stored is as follows.
i). Breakpoint the _l_buf_36() routine, do d *(esp+8) to display in the data window the pointer to the vendor code structure, in some instances you may need to pagein this address via SoftICE.
ii). Set a bpm w on [vendor code structure + 3Ch] and monitor writes, anything other than zero should yield the static location of the data being written there (note that you'll probably get 3 or so breaks on access before finding the right one), once we've located the correct place we can make a small patch of the static data.
:0040100F mov eax, [ebp+1Ch]
:00401012 mov dword ptr [eax], 1 ; Set flag (we want 0)
This requires a patch, in Imaris this flag is AND'd with 0, GerbTool sets it to 1 & SDS to 2. If we make the necessary changes GerbTool will launch and license successfully. I've developed a 'muster' which you might care to print and apply to your next FLEXlm target. Lets apply my technique to SDS 2 v6.318. (Please note that this method should not be totally relied upon as a quick substitute to commenting your IDA database using lmgr.lib), the screenshots below are taken from a fully commented .idb.
* Tip * It can be worthwhile loading lm_crstr.obj in IDA to aid commenting of your target idb.
1. Locate _l_sg().
This seems to be trivial by searching for the hex constant 6F7330B8h (unsigned long x = 0x6f7330b8; /*- v8.x */) used at the start of the routine, (note that there are 2 references to this constant and it is the first one you find which is _l_sg(), the other is inside _l_svk() purely for obfuscation). If you feel so inclined you can label dword_161DEC4 as _l_n36_buff.
_l_sg = sub_DBFD0A
2. Locate _l_init().
_l_sg() should have 6 cross references. By quickly examing the code immediately after each call to _l_sg() looking for the check against the default seeds 0x12345678 & 0x87654321 _l_init() can be identified.
By examining the other references to _l_sg(), _l_good_lic_key() will be the only reference that calls _l_sg() twice, in my fully commented .idb this is clear to see.
_l_good_lic_key = sub_DBE15D
4. _l_good_lic_key() has 9 xrefs of the form 3 xrefs inside 3 functions, the top most of these in the library is _lm_start_real(), again in a fully commented .idb, this is clear to see.
_lm_start_real() = sub_00DBD112
The only xref to _lm_start_real() should be inside _l_checkout().
_l_checkout() = sub_DBCA0E
From _l_checkout() we can easily deduce _lc_checkout() by finding the reference with lm_ckout.c at the start of the function (the other reference to _l_checkout() is inside _l_reconnect()). _lc_checkout() = sub_DBC940
5. _lc_checkout() has 2 cross references, the 2nd of which is of the form mov dword ptr [reg32+30h], offset _lc_checkout() (this is actually part of _InitLmInterface()), by scrolling up here we can label the routine at [reg32+0Ch] (or the first entry) as _lc_init(), this yields _lc_new_job() which call's _lc_init() as sub_DC8610 and _l_n36_buf as dd 0E07100h.
6. In a live debugging session we set breakpoints on :
From the break inside _l_n36_buf() we discover that our 2 bad flags are set at address E07112 & E0D4BD. From the break inside _lc_checkout(), we get the feature name sds2 and the version number 6.318, lets amend the license accordingly.
* Tip * Most FLEXlm targets vendor daemons don't use the _l_n36_buf() configuration flags, this can make them an easy way to recover seeds using lmgrd -z daemon_name -c license file, especially if you don't get a break on _l_sg() in your regular target or _l_checkout() returns EAX=-15.
You should now get a break on _l_sg() and the call to the _l_n36_buff() decoding routine, this should enable you to recover the seeds (0x987AC78F & 0xC8D6382B). Using lmv8gen you can generate vendor keys and now compile lmcrypt.exe. This will enable you to generate a valid license.
* Tip * By leaving your breakpoint on _lc_checkout() active you should be able to recover any other feature names checked out by your target application and thus generate licenses for them.
With SDS 2 the approach I describe above lets _lc_checkout() return 0, however the program crashes unexpectedly afterwards; this was puzzling since the warez released version simply patches sub_DBC940 (what we know to be _lc_checkout() to XOR EAX, EAX / RETN), exactly the same effect as our license. Also the license with the release was evidently generated using the same key information I recovered above (as lmcrypt didn't change it). SDS 2 appears to be an anomalous case amongst FLEXlm targets, I tried 3 others and couldn't replicate this behaviour, however its useful to know that we can also use this somehow more 'brutal technique' of patching _lc_checkout(). Since there is also integrity checking of the file, patches were also made to static checksum data, this was implemented separately by the developers obviously to prevent file tampering.
Conclusion
I hadn't really looked at FLEXlm for quite some time and Macrovision are to be commended for finally making a protection which is safe from key generators, however, whilst they continue to support their legacy mistakes, old style FLEXlm licenses can still be generated and the application trivially patched to make the licensing layer accept them, even if my technique was to be broken, it is still far too easy to patch the _lc_checkout() API directly.
