BBC BASIC for Windows: Utilities

BBC BASIC for Windows
IDE and Compiler >> Integrated Development Environment >> Utilities
http://bb4w.conforums.com/index.cgi?board=ide&action=display&num=1236183192

Utilities
Post by Malcolm on Mar 4^th, 2009, 3:13pm

The reading of the lines from the IDE looked quite simple in the skeleton utility. To write back a modified line would you use the selection (from message 1041) and do a EM_REPLACESEL or is it more complicated?

I was thinking about writing simple Concatenation : Expand formatting and perhaps REM removal kind of operations.

I did not understand the significance of PROC0 but I know you don't do things arbitrarily so it must be important. What is it doing?

Regards, Malcolm.

Re: Utilities
Post by admin on Mar 4^th, 2009, 4:02pm

Quote:

The reading of the lines from the IDE looked quite simple in the skeleton utility.

Er, what "skeleton utility"? Nobody has asked me for the source code yet; I'm expecting a request for it from Michael Hutton at any moment, but nothing received so far. You're welcome to a copy if you have an application in mind.

Quote:

To write back a modified line would you use the selection (from message 1041) and do a EM_REPLACESEL or is it more complicated?

Bear in mind that the BB4W program editor isn't a Windows edit control; it doesn't (in general) respond to EM_ messages, and those few it does respond to often don't work the way you might expect (since BB4W both sends and receives the messages, they are treated as 'private' messages, with custom content, even if they 'borrow' official Windows message numbers).

The 'check quotes' demo utility doesn't need to modify the contents of the edit buffer, but I could in principle generate one that illustrates how to do that. However, actually changing the user's program obviously has ramifications that a 'read only' utility doesn't (should it save the original program, should it change the file name, that sort of thing) so it would require a lot of careful thought.

Quote:

I was thinking about writing simple Concatenation : Expand formatting and perhaps REM removal kind of operations.

Surely 'concatenation' and 'REM removal' are operations that you would want to do as part of the compilation process, rather than to the 'source code'? The last thing I would want is for people to make their programs less readable! If somebody really has a reason to do something like that (and I can't think of one) the way to do it is to extract the output from BB4W's built-in cruncher.

Incidentally I am hoping that Mohsen may want to implement his advanced cruncher (StrongBS) as an 'add on' utility for BB4W. There's not much it can't do, judging by the description of the RISC OS version!

Richard.

Re: Utilities
Post by Malcolm on Mar 4^th, 2009, 5:02pm

We most likely have different ways of going about things, but I write programs first as a series of REMs that describe the logic. Then I fill in the function of that logic with actual code. And within that code I add REM's that describe the things, sub routines, that I have still to code.

What I typically do is put something like a couple of ** into those todo's. I also use the *| REM's to have reminder notes in the code so that I don't get lost quite so often. Then later I go and take them out. It would be nice to do that automatically. Similarly I have had many occasion to take someone elses listing and spread out the multi statement lines so that I can see the indenting. Again that would be a nice automatic function.

Sure it can be done with scanning files and rewriting to a new one. The IDE utilities route seems to be neater.

So I am formally asking whether you would provide the information to allow me to do that.

Malcolm.

Re: Utilities
Post by admin on Mar 4^th, 2009, 9:37pm

Quote:

Then later I go and take them out. It would be nice to do that automatically.

If you use 'regular' REMs, can't you simply use the 'Add REMs' and 'Remove REMs' options already available in the context menu?

Quote:

So I am formally asking whether you would provide the information to allow me to do that.

I'll send you the source of the 'check quotes' utility. As for modifying the contents of the edit buffer from an 'addon' utility, having thought about it I'm not sure there's a satisfactory way of achieving that. Internal to BB4W everything relies on the memory being shared, so I don't have to bother with the 'marshalling' needed to copy strings from one process to another. You possibly could do it by simulating user input (WM_CHAR message), but it would be messy.

Richard.

Re: Utilities
Post by Michael Hutton on Mar 5^th, 2009, 01:29am

Quote:

I'm expecting a request for it from Michael Hutton

Hello. Sorry, up in Queensland at the moment only intermittanly getting to a computer. I have just downloaded the checkquote.exe. I need to go back and see what it does although I can imagine that it checks quotes!

I am working on the principle (after reading this thread) that I can get a program from the IDE line by line using windows messages. This should be easy to add to a utility once it is up an going.

In the meantime I need to fully get to grips with the Varlist207.. which I am doing at present.

My main idea is to get it into a 'decent' GUI such as a listview etc but I would *really* welcome any suggestions on what people would like to see and how people would like to see it. On this note I will start a new thread... Varlist Utitlity here in the IDE section.

On requesting the skeleton code I will email.

Michael

Re: Utilities
Post by Malcolm on Mar 5^th, 2009, 03:29am

Quote:

As for modifying the contents of the edit buffer from an 'addon' utility, having thought about it I'm not sure there's a satisfactory way of achieving that.

So we may as well extract from the IDE, process to a .bbc or .bas and read that back in.

Can the save and read back be remotely triggered? Actually it might be easier just to fire up another copy of the IDE using the file associations.

Malcolm.

Re: Utilities
Post by admin on Mar 5^th, 2009, 08:38am

Quote:

Can the save and read back be remotely triggered?

Goes away to check the BB4W source code....

You could trigger a Save As... operation, but that unavoidably prompts the user for the location and filename, which I guess isn't what you would want. It also changes the 'current' filename to what it has been saved as, which isn't useful.

There's no way of activating the actual 'file write' code in BB4W because that's a straightforward C function call within the IDE; it isn't triggered by a Windows message.

Your best bet may be to read the entire program using the technique in checkquotes.exe and save it yourself. However that might be a little slow for very long programs.

Richard.

Re: Utilities
Post by admin on Mar 5^th, 2009, 09:10am

Quote:

To write back a modified line would you... do a EM_REPLACESEL

Actually that might work, because Windows should do the marshalling for you (since EM_REPLACESEL is a standard message). However you would need to take account of the ways in which the BB4W editor's implementation of EM_REPLACESEL is non-standard:

insert

wParam

An alternative approach would be to put the replacement text on the clipboard, and send a WM_PASTE message. That allows both replacement and insertion (although not deletion) but of course has the disadvantage of destroying the existing clipboard contents.

Richard.

Re: Utilities
Post by Malcolm on Mar 5^th, 2009, 5:21pm

Quote:

Actually that might work, because Windows should do the marshalling for you (since EM_REPLACESEL is a standard message).

I can confirm that it does! That's quite useful.

Malcolm.

Re: Utilities
Post by Malcolm on Mar 7^th, 2009, 2:26pm

Utility Programs that are supplied as an executable cause a warning when used about being unsigned. That stops it from being used 'seamlessly' from the menu. It shows that an .exe is being called and requires authorization in some form.

Is there a way around this other than the authors getting a digital signature , local compilation, or switching the PC security to a low level which may not be within the user's control.

Malcolm.

Re: Utilities
Post by admin on Mar 7^th, 2009, 5:28pm

Quote:

Utility Programs that are supplied as an executable cause a warning when used about being unsigned.

Under normal circumstances you only get a warning about an executable being 'unsigned' if you run it from a web site. I've never seen that warning when executing a file locally on my PC, nor would I expect to.

The 'addon utility' feature in BB4W uses the regular ShellExecute API, so if that is resulting in the warning it's going to affect a great deal more than just BBC BASIC!

Are you sure this isn't a side-effect of a virus scanner or something similar?

Re: Utilities
Post by Malcolm on Mar 7^th, 2009, 6:18pm

You are possible right that it is the virus checker or one of the other security programs that causes this. It certainly looks like a windows warning though. It links to IE and the following text.

Quote:

Should you open files from the Internet or e-mail that you have saved to your computer?
It depends. You should take a few precautions before opening a file that you saved from the Internet or received as an e-mail attachment. If the file is a program or computer code, such as an .exe file, Windows will display a message to alert you that the file could be harmful to your computer. If this happens, ask these questions before you open the file:

Have you scanned the file for viruses?

You should scan all files you download with an up-to-date antivirus program before you open them.

If you do not have antivirus software installed, you should purchase and install an up-to-date antivirus program to help protect your computer. For a list of antivirus programs, see the Microsoft Antivirus Partners (http://www.microsoft.com/security/partners/antivirus.asp) website. If you have already installed antivirus software, make sure that it is up to date and actively scanning the files on your computer. Viruses are released on an almost daily basis, so keeping your antivirus program current is important.

Do you know who created the file?

If you choose to save or open the file, Windows will check the file for a digital signature. A valid digital signature helps to tell you the identity of the publisher (the company, website, or individual distributing the file) and whether the file has been tampered with since it was signed.

If the file has no valid digital signature, you cannot be certain that the file is actually from the source it claims to be from, or that it has not been tampered with. Avoid opening the file unless you have scanned it for viruses and know that the contents are safe.

Do you know what the file will do to your computer?

The publisher of the file should have provided information explaining what this file is for and any special details you need to know about the file to open it. Be sure to read any terms of use or license agreements, and look for anything you don't agree with, such as having to accept unsolicited advertising, or that the program sends information back to the publisher. If this information is not available, you should be cautious about running the file.

Running an executable direct from a web page isn't even an option. I have to download any .exe file to the local machine first. Then I have to 'accept' at this warning. And yes, it affects not only BB4W. Programs that are installed are obviously checked somehow and don't generate the warning, only downloads do.
.exe Attachments to mail are rejected outright.

I guess security here is tight.

Malcolm.

Re: Utilities
Post by admin on Mar 7^th, 2009, 10:25pm

Quote:

You are possible right that it is the virus checker or one of the other security programs that causes this.

I have often found that virus scanners, despite their obvious value, cause unwanted and annoying side-effects (sometimes genuinely in the interests of 'security', sometimes by mistake). Here I use AVG, which not only gets high ratings in comparisons of security products but also seems to be exceptionally free from such side-effects:

http://www.avg.com/

There's a free version (for private use only):

http://www.avg.com/product-avg-anti-virus-free-edition

Richard.

Re: Utilities
Post by Malcolm on Mar 22^nd, 2009, 2:53pm

Thanks, Richard,

Sorry I only just noticed your last post. I actually use AVG so it must be the Windows security level that is at issue.
AVG scanner does mask many of the Icons in the system tray on start up, although this is seen as a Windows 'feature'.

Re: Utilities
Post by Malcolm on Apr 1^st, 2009, 2:04pm

Quote:

Actually that might work, because Windows should do the marshalling for you (since EM_REPLACESEL is a standard message).

One issue that arises is that the screen updates after the replacement as you would expect. I found that doing a large number of replacemnts on a long program causes the program to scroll after each insert as well. This overwhelmed my system. I think the WM_PAINT messages backed up to the point that Windows failed to cope. It caused a black screen and a slow recovery and very slow graphics afterwards or even a Blue screen of death.
Using WM_SETREDRAW message to suppress the redraw to a sensible rate or switching it off temporarily solves the problem. Is there a way to set the IDE's mouse pointer to a busy Icon while it is being manipulated from outside. Setting a messagebox on top of the IDE seemed a bit clumsy.

Malcolm.

Re: Utilities
Post by admin on Apr 1^st, 2009, 10:14pm

Quote:

This overwhelmed my system. I think the WM_PAINT messages backed up to the point that Windows failed to cope. It caused a black screen and a slow recovery and very slow graphics afterwards or even a Blue screen of death.

I'm somewhat surprised the consequences were that severe; it shouldn't be possible to provoke a BSOD by that means. Have you tried it on a different PC, just in case you've revealed a weakness in the particular machine on which it's running?

Quote:

Is there a way to set the IDE's mouse pointer to a busy Icon while it is being manipulated from outside.

Setting the cursor shape is easy enough (SetCursor API) but the trouble is as soon as the user moves the mouse it will revert to what it was previously. You could try calling SetCursor so frequently that it is predominantly IDC_WAIT but it would be bound to flicker when the user moves the mouse.

AFAIK there's no method of changing the cursor shape to IDC_WAIT that would cause it to stay that way when the mouse is moved. However you can change the cursor to IDC_NO (the 'no entry' sign) if that would be an acceptable alternative:

Code:

 SYS "SendMessage", hEditor%, 1032, 0, 0

To set it back to normal:

Code:

 SYS "SendMessage", hEditor%, 1032, 1, 0

Richard.

Re: Utilities
Post by Malcolm on Apr 26^th, 2009, 4:11pm

With regard to Utilities I have two questions.

Does anyone have any thoughts about where the best place to locate them is?
How can I tell if the IDE that I want to operate on is actively running a program.

regards,

Malcolm

Re: Utilities
Post by admin on Apr 26^th, 2009, 5:55pm

Quote:

Does anyone have any thoughts about where the best place to locate them is?

I've created a UTILITIES subdirectory in the main BB4W installation folder (typically C:\Program Files\BBC BASIC for Windows). However that's just my personal preference, and of course Admin rights are required to put files there (that seems correct for an 'extension' to BB4W, but not everybody may agree).

Quote:

How can I tell if the IDE that I want to operate on is actively running a program

That isn't straightforward, but it's very easy to discover whether the 'output window' is open (generally that's what you want to know, because if the interpreter is sitting at the command prompt it's the same as running a program for most practical purposes).

To find out whether the output window is open simply use the 'IsWindow' API on its handle (the third and last optional parameter that can be passed to a utility).

Richard.

Re: Utilities
Post by MDCHutton on Apr 26^th, 2009, 5:58pm

>How can I tell if the IDE that I want to operate on is
> actively running a program.

From ' inside' the runing program, Yes. (I can't see your post in this page.) if your in a TimerProc (a callback procedure made with a timer) you can look at what the statement pointer is looking at and if it is 0, there is no statement there and hence you are in immediate mode! If your in an INPUT, GET or INKEY... it will give you the token for the relative one. You can also test for END and QUIT the same way (although in PROFILERvx.x I change all the QUITs to END before running the program because it leads to faster TimerProc..You could theoretically test for any keyword but I can only think the TimerProc would become to unwieldy...

but from 'outside', I don't know.

Michael

Re: Utilities
Post by MDCHutton on Apr 26^th, 2009, 6:03pm

Sorry, on the topic of where to put them - in their own folder under a \Utilities sub folder of \BBC Basic for Windows because of issues of removing them. RMDIR /s /q is an simpler option than relying on the user to 'register' every file they add.

Re: Utilities
Post by MDCHutton on Apr 26^th, 2009, 6:10pm

on Apr 26^th, 2009, 5:55pm, Richard Russell wrote:

and of course Admin rights are required to put files there (that seems correct for an 'extension' to BB4W, but not everybody may agree).

I try to write a {GUID}.txt file to the directory the user has chosen and if they can't, they don't have access rights, and they have to choose again. I think it's a Vista drama? I would like custom filter the BrowseFor folder for access rights...but as to a default folder. under BB4W would be good.

Michael

Re: Utilities
Post by admin on Apr 26^th, 2009, 9:25pm

Quote:

I think it's a Vista drama?

No, that's a misapprehension. It applies equally to all versions of Windows NT, including Win2K and WinXP. The only reason Vista comes into it is that (thank goodness) it actually encourages running not as an administrator, so you're somewhat more likely to encounter 'permission' issues on that OS.

People running XP as a 'normal user' (which they always should, but most don't) will have similar issues with writing to Program Files. But of course that's exactly as it should be - an ordinary user ought not to be able to upset a 'shared' application.

Arguably, since the Registry entries controlling the add-in utilities, are 'per user' (i.e. in the HKEY_CURRENT_USER hive) the utilities themselves ought also to be stored in a 'per user' location. So maybe they should go in 'Documents and Settings\[user]\Application Data'.

Richard.

Re: Utilities
Post by Malcolm on Apr 26^th, 2009, 11:18pm

Thanks, Richard,
The output window is not going to distinguish between running and having completed necessarily.
No, I really was trying to find out if it had ended, if it ever was run, as I did not want any chance of writing to a program that was still running and thus crashing it, which seemed likely. I was hoping there would be a flag I could look at.

Quote:

(the third and last optional parameter that can be passed to a utility).

There seemed to be other data on the command line sometimes. Are they then "garbage"? I often see a '10' for a 4th parameter but am I just picking up random dross.

Malcolm.

Re: Utilities
Post by admin on Apr 27^th, 2009, 08:50am

Quote:

I did not want any chance of writing to a program that was still running and thus crashing it, which seemed likely.

Just because the interpreter isn't actually 'running' a program doesn't mean you can't crash it! If you change the current program 'behind the interpreter's back' (i.e. change the contents of memory starting at PAGE) something as simple as an immediate-mode command entered at the prompt could crash it. That could happen, for example, if you increase the length of the program - since it will then overwrite the beginning of the heap (or the heap could overwrite the end of the program).

From the point of view of an add-in utility, the interpreter sitting at the command prompt is no different from the interpreter waiting for the user to respond to an INPUT statement. In neither case must you modify the program in memory.

So, whether the output window is open is exactly what you need to know to determine whether it's safe to modify the program. Indeed, that's why the IDE itself uses that information to enable or disable editing - if it's right for the IDE it must be right for a utility!

Quote:

There seemed to be other data on the command line sometimes. Are they then "garbage"? I often see a '10' for a 4th parameter but am I just picking up random dross.

They're not "garbage" or "dross", they are command line parameters which you have provided (accidentally or deliberately). If you see something unexpected in the command line, check what you've stored in the Registry as the 'Parm' entry for that utility. I'm willing to bet you'll see the data there!

(Edit: alternatively, if you forget to store in the Registry the terminating NUL of your Parm string, you could be seeing uninitialised memory contents. Ensure you use code similar to that in 'checkquotes.bbc' to write strings to the registry).

You can use the same 'utility' executable to provide multiple entries in the Utilities menu, by adding your own command-line parameter(s) in the Parm data so that the executable knows which menu item was activated. That way you can package multiple utilities together in a single EXE if you wish.

Richard.

Re: Utilities
Post by admin on Apr 28^th, 2009, 08:23am

Quote:

if your in a TimerProc (a callback procedure made with a timer) you can look at what the statement pointer is looking at and if it is 0, there is no statement there and hence you are in immediate mode!

I've no idea where you got that from, but it's not correct. To test for immediate mode you can check whether the statement pointer is < PAGE, but you can't tell (reliably) by looking at what it is pointing to. Even the '< PAGE' test could be fooled if PAGE has been raised by a program.

Also note that the 'statement pointer' isn't guaranteed to be pointing to the start of a statement. In some circumstances it can point part way through a statement. For this reason be careful how you use it.

Richard.

Re: Utilities
Post by Michael Hutton on Apr 28^th, 2009, 1:21pm

I should have been a bit more explicit. You look at the byte that the statement pointer is looking at and that seems to normally be the keyword token. So far in my limited tests when the program has run through to immediate mode the byte the statement pointer is addressing seems to be 0.

So in Profiler this seems to work:

Code:

      
        .TimerProc
          
         mov dword [^@%+392],!(^@%+392)      
         mov eax,[^@%+384]   
          
          cmp byte [eax],&E0  ;look for END
          je close
          
          cmp byte [eax],0      ;if nothing there ? immediate mode
          jz close
          
        
....etc

So far I have got profiler to reliably exit when in immediate mode....

Michael

Re: Utilities
Post by admin on Apr 28^th, 2009, 2:52pm

Quote:

when the program has run through to immediate mode the byte the statement pointer is addressing seems to be 0

I understood perfectly well what you meant! I have (again) looked at my code and I am certain that it is not a reliable test for being in immediate mode.

As far as I can see, if the interpreter enters immediate mode by 'falling off' the end of the program (i.e. by detecting a line-length byte of zero, being the first byte of the 00 FF FF terminating sequence) the 'statement pointer' ends up addressing the byte stored at TOP, i.e. the byte following the end of the program.

If that is the case, you are relying on the byte stored at that location (the first byte in the heap) being zero, which most definitely is not guaranteed, and is not even particularly likely. I reckon it's pure luck that you have found it to work.

Richard.

Re: Utilities
Post by Michael Hutton on Apr 29^th, 2009, 12:46am

Ah. I will test that one a bit more then, and see how many times I can be 'lucky'. I had only used 3 or four programs to test out the idea and I seem to have been lucky so far... I think I had assumed it was seeing the last 00 FF FF and ending there.

Thanks for the update and thoughts on that one.

Michael

Re: Utilities
Post by admin on May 7^th, 2009, 9:34pm

In all normal situations it is not necessary to modify @lib$ in your programs; indeed all the 'system string variables' (@dir$, @lib$, @tmp$, @usr$) should be considered 'read only' values.

It is particularly important not to modify @lib$ because doing so in a 'compiled' BB4W program can - if you're unlucky - be highly destructive of disk contents. This is because of the way BB4W executables 'clean up after themselves' on termination.

Unfortunately the special requirements of 'add-in utilities' mean that modifying the value oif @lib$ can be necessary. I had, until today, believed that there was a 'safe' way to do so, but it turns out that I was wrong.

I have therefore created a special version of BBCWRUN.EXE which can be used when compiling such utilities. It still requires special measures to be taken when modifying @lib$ (you must not use a simple assignment statement) but when those precautions are taken the risk of accidentally deleting wanted files is eliminated.

If anybody is working on a program that needs to modify @lib$ please contact me privately so that I can supply you with a copy of the special version of BBCWRUN.EXE and explain how, using it, @lib$ can be safely modified.

Richard.

Re: Utilities
Post by Malcolm on May 10^th, 2009, 01:14am

With regard to the recent thread in Yahoo I can see the problem with insertion at the caret position. The question is if you did want to insert something at the caret (in a utility) how would you do that?

If you have a selection highlighted then that's pretty easy but otherwise?

EM_GETSEL seems to need some movement of the mouse to register a valid position even if no highlighted selection is apparent, and just gives the line when there is a selection.
The Status line shows the info but how can we obtain that location?

I tried reading the status line but that did not seem to work although I located the status window ok.

Re: Utilities
Post by admin on May 10^th, 2009, 10:06am

Quote:

if you did want to insert something at the caret (in a utility) how would you do that?

To find the caret position you can send message 1034 (WM_USER+10). The value returned is (yCaret << 12) + xCaret

Richard.

Re: Utilities
Post by Malcolm on May 10^th, 2009, 11:55pm

Quote:

To find the caret position you can send message 1034 (WM_USER+10).

Yes, that works nicely, Thanks again Richard.

Regards, Malcolm.

Re: Utilities
Post by Michael Hutton on May 28^th, 2009, 08:32am

I was trying to detect when a new line has been added to the IDE either (I thought) was to:

1. Send a EM_GETLINECOUNT message with a certain delay eg 50ms or so but this can cause problems (ie all lines are reformatted and any the characters in the first 5 spaces will get shifted along the line). I imagine that EM_GETLINECOUNT also triggers the IDE to format the IDE into a recognisable form.

2. Try to PeekMessage the IDE message queue and intercept and WM_CHAR messages with the value of &D and then read the line, but I can't PeekMessage another thread's message queue.

Is there a way of detecting when the 'Enter' key is pressed in he IDE? The idea is to have a utility automatically get a new line rather than having the user to trigger a --do this-- utility.

would be great for the WINCONST utility....

Michael

Re: Utilities
Post by admin on May 28^th, 2009, 10:37am

Quote:

intercept and WM_CHAR messages with the value of &D

I suppose in principle you could do that by subclassing the editor window, but the potential for getting it wrong and crashing out the editor is such that it's not something I would want to recommend.

Richard.