LMD-ScriptPack: Access Violation in ConvToVar when creating new objects

Konzeptum GmbH

In our application we want to execute scripts on a list of objects (several hundred to thousand items). For each object one script will be executed, the scripts are similiar but differ from each other. We use always the same LMDScriptControl-instance and process the scripts sequentially.

Our script-code looks fine and works on single execution. But if we process the complete list some script-executions result in an Access Violation. On average, 5% of the cases are erroneous. If we try to rerun the erroneous scripts, they complete successfully.

Until now we have not been able to create a simple example application, which makes the error reproducable. However, with the help of the LMD source code we could probably already identify the source of the problem (and possibly even find an aproach for the solution).

Our Script-Code looks like this:

var
  myObject;
begin
   myObject := myController.CreateTestInstance('TTestSubClass');
  try

     //Do something

     myObject.SomeFunction();
  finally
     myObject.Free();
  end;
end;

This is a simplified Code-example of how we run the scripts in the delphi-application

procedure TForm1.bExecuteClick(Sender: TObject);
var
  controllerObject : TControllerClass;
begin
  try
    controllerObject := nil;

    var amountStr : string := eAmount.Text;
    for var i := 1 to amountStr.ToInteger() do
      begin
        controllerObject := TControllerClass.Create();

        LMDScriptControl1.Language := slPascal;
        LMDScriptControl1.Source.Clear();
        LMDScriptControl1.Source.Text := eScript.Lines.Text;
        LMDScriptControl1.Prepare();
        LMDScriptControl1.AddUnits([UTestClasses_sw], true);
        LMDScriptControl1.AddObject('myController',
                                    TControllerClass_sw.ToVar(controllerObject));
        LMDScriptControl1.Open();
        LMDScriptControl1.Close(true);
        FreeAndNil(controllerObject);
      end;
  except
    on e : Exception do
      begin
        //not active but still holding Objects, if Open() fails
        LMDScriptControl1.ClearObjects();
        FreeAndNil(controllerObject);
        lStatus.Caption := 'Error.';
        MessageDlg(e.Message, mtError, [mbOk], 0);
      end;
  end;
end;

We pass an object via AddObject to the script ("myController"). Each script will get an individual instance which will also be freed after the execution.

In the script we create a new Object with the given object. We do not call the contstructor directly, instead we have a factory method. The method creates an Object of a class, which has a LMD-ScriptWrapper (TTestClass). But the method may create an instance of a subclass (TTestSubClass), which has no individual ScriptWrapper-Class.

The call of CreateTestInstance throws the exception.

Sorry for the long introduction, but we wanted to provide as much background information as possible. In the following I describe the specific problem and our analysis.

The wrapped code of CreateTestInstance runs normally, the new instance is created. The call ToVar / ConvToVar in the wrapper results in the access violation:

function TControllerClass_CreateTestInstance_si(var AObj;
  const AArgs: TLMDDispArgs; AArgsSize: Integer; IsGet: Boolean): OleVariant;
begin
  Result := TTestClass_sw.ToVar(TControllerClass(AObj).CreateTestInstance(
    string(AArgs[0])));
end;

We have dived in a little bit deeper in the source code and assume to have found the problematic code.

class function TLMDClassWrapper.ConvToVar(AObject: TObject): OleVariant;
var
  wrrcls:  TLMDClassWrapperClass;
  wrapper: TLMDClassWrapper;
begin
  if AObject = nil then
  begin
    Result := LMDVarNil;
    Exit;
  end;

  wrapper := TLMDClassWrapper(WrapperByObject.Find(AObject));
  if wrapper = nil then
  begin
    wrrcls  := TWrapperClassItem.Get(AObject.ClassType);
    wrapper := wrrcls.Create(AObject);
  end;

  Result := WrapperToVariant(wrapper);
end;

ConvToVar first tries to find the object in its hashtable. If the object is not known, a new wrapper-object will be created. Finally the wrapper will be returned as variant.

Since we have just created the instance (AObject), this cannot be in the hashtable yet. But in some cases the Find will return a result. But the returned object does not appear to be valid and seems corrupted, the WrapperToVariant call will then raise the exception.

We assume, that there is some problem with the memory management. It seems that the HashTable holds objects which were already destroyed, and under certain circumstances returns that object-pointer by mistake.

As mentioned at the beginning, we tried to reproduce this in a simple sample application, with the code above, but we were unable to do so. The sample application did not produce any access violations.

We enabled some logging in the lmd components and could observe an interesting behavior in our application after the access violation. After execution every minute were several calls to the HashTable.Find method made. We assume that there ist some backgroundthread which tries to clean up the HashTable. Maybe in some cases the data is so much corrupted that it cannot complete the tidy-up-process.

We made a simple adjustment in the source code and could actually run thousands of scripts without error. We added a global unit-level function
to LMDSctWrappers and call this after every script-execution (After
LMDScriptControl1.Close(true)). We assume that this cause some side-effects but it may help you to identify the cause of the problem and find a solution.

procedure HotfixClear();
begin
  WrapperByObject.FMap.Clear();
end;

Eugene Balabuev (LMD)

Thanks for your long description. Can you disable the following project's option:

Delphi Compiler -> Linking -> Support address space layout randomization (ASLR)

Let me know, whether this will help...

> We assume that there ist some backgroundthread which tries to clean up the HashTable.

No. Script engine needs to know when any of Delphi object is destroyed. This is a big issue, and we intercept memory manager calls, like GetMem/FreeMem. But, since this should not introduce a major slowdown, we keep the tricky bit-based memory map, which allows to optimize things to acceptable level.

This map, however, uses a high bit (or two - I don't quite remeber) of Pointers, which was fine in previous versions of Windows/Delphi. Anyway, this is not easy to fix, but probably need to be reinvestigated.

Konzeptum GmbH

Hello Eugene, thanks for the fast reply.

The ASLR was also one of our ideas and we already tried it. Changing the setting made no difference.

In our environment, we can reproduce the error very well.

If you need more information, e.g. states of certain variables at runtime, or would like to have a look at the problem in a video call, feel free to let us know.

We have encountered the problem on two different environments yet:

Embarcadero® Delphi 11 Version 28.0.47991.2819
Windows Server 2016 Datacenter Version 1607

Embarcadero® Delphi 10.4 Version 27.0.40680.4203
Windows Server 2019 Version 1809

Eugene Balabuev (LMD)

Need to think...

Konzeptum GmbH

Hello Eugene,
are there any new findings on this issue yet? For us, this issue is relatively critical, as our customers want to run scripts in large quantities, but due to the error there is a high scrap rate. A temporary workaround would also help us a lot at first.
As last mentioned we are ready to work together to find the problem and the solution. For example, we could do a debugging session together where we provoke the error in our environment.
If this would help you, or you have another idea on how we can help, please feel free to let us know.

Eugene Balabuev (LMD)

Fix is provided via e-mail.

Konzeptum GmbH

I have downloaded the latest release and the fix works fine. Since then no more problems.

Thank you very much for the quick solution.