The Ashley Book of Knots INDEX Project, 2008 - IT issues

[Sweeney]

I can appreciate that a lot of work has gone into this but if the object is to index Ashley completely then forward references should fall out as knots are added. For example there is no need to say which knots are started with a RT as this would be referenced when these knots are indexed - to arrive at the list above one would simply search this field. Again I strongly recommend designing the method of storing and retrieving this data before going any further. Happy to help.

Barry

[PwH]

Barry,

that sounds like a very good idea, but I have no idea how to do it nor the time to learn. Will you do this? I will give you all the information I have to facilitate this.

  Cheers, Peter

[Sweeney]

Hi Peter

I suggest we collaborate  - there is not much to learn to get the data recorded - but do you have a copy of MS Excel? I'll happily design a sheet to get going and perhaps we can agree (with others joining in if they wish) how best to divide the work. Once we are well underway I suggest asking for comments on the methodology as once we've got that about right not only can the job proceed apace but verifying and editing can also get under way. Incidentally Excel is not essential - a word processor would do and the final form of the index can be agreed once the basic work is nearing completion (I have in mind that this lends itself to a computer solution not a printed one). To start with I'll put some knots into a s/sheet to demonstrate and test - this I can email to you and post a text/pdf version but the latter would have limited search capability.

Barry

[DerekSmith]

If you do not have MS Excel, could I suggest that you work with OpenOffice which works with the same files but is totally free.  I use it for all my spreadsheet and word processor work and it can export/import in MS format if you want.

As for a database, we are probably not too far away from the Guild waking up fully to the importance of the Internet, and with this realisation will come the opportunity of having Guild based databases served by SQL and PHP so that Guild users can carry out searches through the Guild website.  This has the advantage that each user does not have to carry an up-to-date copy of the database on their computer and they would always be accessing the most uptodate version whenever they did a search through the website.

If you are thinking of this sort of usability, then perhaps inviting Mel to help with the setup might help in the future.

Derek

http://openoffice.org-2009.com/index.asp?aff=101&camp=gg_oo_uk&se=google

[Sweeney]

Derek

I agree - to start with I'd like to try out a simple flat file in Excel to get a better idea of how to record the basic info.

Mel

Any help appreciated, please.

Barry

[Knot Head]

That wont be too hard to do. As you can most likely tell this Forum board is built on php with mysql on the backside holding the data stored from all the posts. My website is based on php and mysql... If you were to build this project using both of these fabulous web programming languages, you would be able to build the web data base program in a fair amount of time and then populate the data base much quicker.

While building the initial data base in a flat file would give you a better idea on how to build the data base structure in mysql, I would not recommend using a flat file for the sole purpose of the data base itself. Excel will not handle that much data being served up on a server that will most likely have a very large scale amount of users accessing it continually 24/7. The load on the server would be greatly diminished. When the server starts to experience a heavy load, then the operations within the server will slow down producing a very slow response time across the DNS. As you will be able to tell with building the flat file data base, that after it starts to get full, even on its own it will take some time to start up just by itself.

Using php and mysql is the way to go for such a task as you all speak of. I love programming with both of them when it comes to building websites and ecommerce solutions. Both solutions can be secured using the HTTPS for secure logins and for secure site roaming and administrations. Any data that sits behind the HTTPS is encrypted at the 64 bit level as a start. Say you open a little ecommerce end, or you are just surfing the site and the many different pages, all that is encrypted. Every piece of data that is sent out from behind the HTTPS stays encrypted until it reaches for example your credit card processing company that then has a key on their side to decrypt the data sent to them. Just another really great feature of php and mysql working together as a team.

Well just my 2 pennies worth as from my own experience. If you were to build this application, php and mysql is the way to go. I have been using both for years now and stand behind it 100%...

Brian...

[PwH]

Hi Barry, Derek, Brian and others,

Yes I do have Excel, so no problem there, Also have Access which I believe is a database app although it looks like a fancy version of Excel to me? Would that be any better to use for a pilot?

Good to see a general level of agreement on the way to go forward, lets go for it!

Cheers, Peter H

[DerekSmith]

Brian,

How might a group of volunteers, not conversant with the technicalities of php and sql to your level, best proceed ?  Or would everyone need to learn these systems?

Derek

[Sweeney]

Brian

Am I right in thinking that SQL and PHP would, in an application like this, be used to front end a data file? If so are there any particular constraints on the file format? Excel is an easy to use tool for gathering data which can then be transferred to virtually any format (if necessary via an intermediate transfer format) such as dBase for example. I don't see a need for a relational database although Access could be used if that simplifies things.

Barry

[Sweeney]

This is a call for volunteers to help with the Index project. I have developed an Excel spreadsheet to record basic details about all of the knots in ABOK (with the possible exception of some of the designs using multiple knots) but including every number, knot or not  and simply showing fastenings which are not knots (in a minute I'm going to get the k and the n the wrong way round!) as just that. The sheet uses a variety of codes to save typing - these can be replaced in the final version with text.  To convert to MySql should be straightforward but Excel provides a good editing tool for now. Notes are included on a separate sheet and comments are there to help. No experience of Excel beyond being able to type in columns is needed (and if you don't have Excel then another program such as Open Office is just as good.

I will co-ordinate who is doing what using a control sheet and once we have finished a substantial chunk I will ask Mel to put the sheet on the website for downloading, commentary (always welcome anyway) and error correction. More about that later. I think that up to 6 volunteers initially and let's see how we get on (more than that once I have made sure I can cope). Rather than post on the forum please email me (barry@lizziemint.co.uk) and I'll send you a blank sheet for completion. Please let me know whch Chapter you would like to undertake (I suggest sticking to one at first) - Chapters 3 and 4 I have done already and Chapters 1 and 2 I would leave until last as the knots there are a repeat of knots covered in later chapters (the sheet cross refers to the prime number for a duplicate knot rather than record the detail twice or more - the prime number is the one in the Chapter dealing with that type of knot eg Bends).

So first come first served drop me an email with the Chapter you would like to do and I'll take it from there.

Barry

[Sweeney]

No volunteers so far - has everybody lost interest?

[admin]

If the initial data is gathered using Excel, I can upload it into a MySQL table using something called a cvs import to get you going. Once the initial data has been assembled, it can also be examined with a view to the best way to store it in MySQL. I appreciate that a flat file might, at first, seem to be the best way to proceed. And it's certainly the simplest approach. However, if this is truly intended to be a long term project with the capacity to handle a whole range of searches and queries, then a series of relational tables right from the start might be the best way to future proof the whole thing.

(For the less technically-minded, relational tables are simply a way of storing, for example, the ABOK reference name of a particular knot in one place, the information on its construction in another, it's potential uses in a 3rd table etc etc. Splitting the data up in this fashion greatly reduces the amount of work that the server needs to do to provide answers over a whole range of questions)

Once the data was imported into MySQL, attention would need to switch away from Excel and over to the central online system for all additions and amendments. Initially I could either write, or source, a basic front end system for all of the data management - ideally accessed by authorised individuals only using a username/password combination. Longer term, a more sophisticated admin interface could be gradually added. If this was tackled on a modular basis, it would limit any additional expenditure from the Guild's point of view and go some way to ensuring that the right tools were being built in the most appropriate way.

As soon as the data is online, work could also begin to allow users to run searches etc - either on the public site or within a restricted area.

Someone mentioned using Access. Frankly, if the intent is to get this online asap, I wouldn't bother. Access isn't well suited for online storage or usage - especially on the igkt.net server. Stick with Excel.

Ultimately, a full centralised, dynamic, online system is highly feasible. However, until I see some sample data, I can't really advise as to how it should eventually be organised other than to echo Knot Head's suggestions that a combination of PHP and MySql would be the way to go.

[Sweeney]

Many thanks for the reply Mel.  So far I have only one volunteer (Squarerigger who is looking at Chapter 40 - probably the biggest job of all so I'm grateful) but I've completed 4 chapters so this should give you an idea of the data being recorded. I'll email you a copy of the spreadsheet to have a look at for now - if it's worthwhile splitting this into related tables then I should be able to easily do this in Excel for uploading - at first glance this might be a starter where a knot has several names or is duplicated - can't think of any other one-to-many relationships but I'll think about it. If anyone is interested in looking at the data rather than doing any recording let me know.

Barry

[admin]

It's been suggested that the latter part of this thread might be better placed in the Computing board to allow technical discussion centring around IT to continue without causing aggravation to other posters.

Does anyone object?

Are there any suggestions for a suitable 'split point' where the discussion becomes more IT than knots?

Mel

[Sweeney]

I suggest from reply #46 onward - I think that was when I started the techie discussion.