why whouldn't use the sqlite to stored file index?

May 8, 2009 at 9:10 AM

implement file store index was quickly growing large.i wouldn't use defraqqing to reduce index file size,because it need to work with sql

Coordinator
May 9, 2009 at 9:42 AM

Hello zhengchun,

Regarding the file store index size;

I fully agree with you about the quickly growing (see the other wiki pages I made), if you use 'random' GUIDs, or 'Combined GUIDs' the index file grows way too big too fast. I therefore suggest to use either a customer function to generate a GUID, or to use either the extension method I created to the String class (see the namespace namespace FileStorage.Helper). In the upcoming version of NFileStorage I also added these helper functions to generate the GUIDs based upon a timestamp and datetime tick; maybe you will also find them useful;

  • public static Guid CreateDataIdentifierByDateTimeUtcTimeTicks()
  • public static Guid CreateDataIdentifierByReadableDateTimeUtc()

If you use this approach the index file does not grow as if it exploded, but in a more 'nicer' way.

Once you have a big index file, the defragging feature might be of help (I used it myself) even though I had references in my SQL table pointing to the items in the index file; the defragging feature also generates a 'sql update script' that will update your SQL tables to point to the correct defragged indexes.

Also one of the things I have in my mind for some time now, is to make the index structure pluggable; so the actual structure of the index file could be implemented in various ways, each with its pro's and con's.

Regarding your initial question; why not use sqlite to stored file index;

To me its not clear whether you mean to use SQLite rather than NFileStorage (as a replacement), or whether you suggest to sotre the index file in SQLite rather than in the format its currently stored in (I think you mean the first one). I can imagine SQLite is better indeed in some scenario's (its listed as one of the alternatives on the homepage of NFileStorage); I simply have not yet tried it before (I only found out about that alternative after NFileStorage was made). Since I don't know it I can't compare the two, but I think a benefit of NFileStorage could be that its up and running in no-time, and you simply have a simplistic C# API with a decent set of features to access a serialized dictionary. SQLite is way more than just a serialized dictionary; its a full database engine (with is definately a plus ofcourse) but for my case - where I only want to store some pictures and access them - the NFileStorage covers it all.

Thanks for sharing, and let me know if you know of other pro's con's worth to others!

------------------------------------------

See the lines below of the new helper class which you could use to produces other GUIDs based upon timestamps and strings;

 

using System;

using System.Collections.Generic;

using System.Linq;

using System.Net.Mime;

using System.Text;

 

namespace FileStorage.Helper

{

    /// <summary>

    /// Helper class that exposes functionality to generate, convert or create data identifiers.

    /// </summary>

    public static class DataIdentifierHelper

    {

        /// <summary>

        /// Returns a byte array of a specified string

        /// </summary>

        /// <param name="text">The text to go into the byte array</param>

        /// <returns>A byte array of text</returns>

        private static byte[] GetBytes(string text)

        {

            return ASCIIEncoding.UTF8.GetBytes(text);

        }

 

        public static string ToNFileStorageOrigFileName(this Guid dataIdentifier, bool suppressNonReadableCharacters, char placeholderForUnreadableChars)

        {

            string result;

 

            try

            {

                byte[] bytes = dataIdentifier.ToByteArray();

                var encoding = new ASCIIEncoding();

                result = encoding.GetString(bytes);

                if (suppressNonReadableCharacters)

                {

                    //

                    // only return chars 32 - 127

                    //

                    var tempResult = new StringBuilder();

                    foreach (var c in result)

                    {

                        if (c >= 32 && c <= 127)

                        {

                            tempResult.Append(c);

                        }

                        else

                        {

                            tempResult.Append(placeholderForUnreadableChars);

                        }

                    }

                    result = tempResult.ToString();

                }

            }

            catch (Exception)

            {

                result = "****************";

            }

 

            return result;

        }

 

        public static string ToNFileStorageOrigFileName(this Guid dataIdentifier)

        {

            char placeholder = '*';

            string result = ToNFileStorageOrigFileName(dataIdentifier, true, placeholder);

            if (result.Contains(placeholder))

            {

                result = "****************";

            }

            return result;

        }

 

        public static Guid ToNFileStorageDataIdentifier(this string text)

        {

            return ToNFileStorageDataIdentifier(text, true, true, true, '_');

        }

 

        public static Guid ToNFileStorageDataIdentifier(this string text, bool autoPadLeft, bool autoTakeFirst16Chars, bool caseInsensitive, char padLeftChar)

        {

            if (autoPadLeft)

            {

                text = text.PadLeft(16, padLeftChar);

            }

            if (caseInsensitive)

            {

                text = text.ToLowerInvariant();

            }

 

            byte[] bytes = GetBytes(text);

            if (autoTakeFirst16Chars)

            {

                bytes = bytes.Take(16).ToArray();

            }

 

            int length = bytes.Length;

 

            if (length > 16)

            {

                throw new Exception(string.Format("The text {0} you provided cannot be used for a dataIdentifier; its too big (resulting in {1} bytes)", text, length));

            }

            if (length < 16)

            {

                throw new Exception(string.Format("The text {0} you provided cannot be used for a dataIdentifier; its too small (resulting in {1} bytes). Consider appending spaces?", text, length));

            }

 

            return new Guid(bytes);

        }

 

        /// <summary>

        /// Returns a GUID based upon a datetime utc format in ticks

        /// </summary>

        public static Guid CreateDataIdentifierByDateTimeUtcTimeTicks()

        {

            Guid result;

 

            result = DateTime.UtcNow.Ticks.ToString().ToNFileStorageDataIdentifier();

 

            return result;

        }

 

        /// <summary>

        /// Returns a GUID based upon a datetime utc format (yyyymmddhhmmssmmmm)

        /// </summary>

        public static Guid CreateDataIdentifierByReadableDateTimeUtc()

        {

            Guid result;

 

            var now = DateTime.UtcNow;

 

            var timestamp = new StringBuilder();

 

            var year = now.Year.ToString().PadLeft(4, '0');

            timestamp.Append(year);

 

            var month = now.Month.ToString().PadLeft(2, '0');

            timestamp.Append(month);

 

            var day = now.Day.ToString().PadLeft(2, '0');

            timestamp.Append(day);

 

            var hour = now.Hour.ToString().PadLeft(2, '0');

            timestamp.Append(hour);

 

            var minute = now.Minute.ToString().PadLeft(2, '0');

            timestamp.Append(minute);

 

            var secs = now.Second.ToString().PadLeft(2, '0');

            timestamp.Append(secs);

 

            var msecs = now.Millisecond.ToString().PadLeft(4, '0');

            timestamp.Append(msecs);

 

            result = timestamp.ToString().ToNFileStorageDataIdentifier();

 

            return result;

        }

    }

}


May 9, 2009 at 3:39 PM

barkgj,thank you for your response.

i'm agree your opinions,Specially about make the index structure pluggable,then we had more choice. i'm looking forward :)