Firebird bug CORE-5452 is fixed : Segfault when engine's dynamic library is unloaded right after closing worker threads

Firebird bug CORE-5452 is fixed http://tracker.firebirdsql.org/browse/CORE-5452

The issue was reported multiple times in Firebird devel list, I will mention here Damyan Ivanov (Debian) and Stephan Bergmann. (RedHat)

And related commit in Firebird 3.0 branch

https://github.com/FirebirdSQL/firebird/commit/40f782ae3e918c4f3842571ff8064be1c4f54961

Stephan Bergmann contributed the patch to LibreOffice in external/firebird

LibreOffice bug #103074 : Implement Boolean Type for Firebird driver is now fixed

Bunth Tamás committed a patch related to issue #103074 : Implement Boolean Type for FB driver .
It has been pushed to "master":

It will be available in 5.4.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.

ps:
From the list a of Firebird integration blockers there are only a few left

Listed blockers:

   70425: EDITING: Embedded Firebird - Copying integer values from Calc to Base impossible  See dependency tree for bug 70425
   70506: EDITING: Embedded Firebird - Alias in queries are ignored  See dependency tree for bug 70506
   104316: FIREBIRD 3 : attempting to open a Firebird 3 embedded ODB database file fails to draw all UI elements, causes general LibreOffice UI malfunction

Firebird 3 by default in LibreOffice 5.4

Lots of missing features & big bugs were fixed recently .
All of the blockers that were initially mentioned on tracking bug are now fixed.

Lionel mentioned that he added a few more blockers that needs to be fixed until Firebird will be switched by default :

I went over
https://bugs.documentfoundation.org/buglist.cgi?quicksearch=firebird

and added a few open bugs to the blocker bug
https://bugs.documentfoundation.org/51780

based on their description (and NEW instead of UNCONFIRMED status)
only, without reading the bug log in great detail.

Tamás Bunth (Wastack) agreed that 6 months time frame is more than enough to fix bugs that will appear

I agree, it would be reasonable to wait a half year. Until that we can
fix the already found and the upcoming bugs, solve the backward
incompatibility. Even if those can be solved quickly we can upgrade to
the c++ API. 

Another issue is importing old hsqldb databases without java installed

It requires making a read-only parser for the HSQLDB "cached" table
data format in another language than Java. Most definitely possible,
the format may even be better documented than "read the source code".

Service for migrating Firebird 2.5 format to Firebird 3.0

On Libreoffice Devel there is a thread about migration of the already created odb files with 2.5 databases inside (fdb) but also about the forward and backward compatibility of the Firebird backup archive (fbk).
https://lists.freedesktop.org/archives/libreoffice/2016-November/075953.html

Lionel added notes about upgrading in the wiki :

People trade privacy for convenience or the other way round all the time. People that in this case favor convenience can use the hypothetical web service. People that favor privacy can follow the instructions at
https://wiki.documentfoundation.org/Development/Base/FirebirdSQL#Older_Firebird_databases(which the web service should point towards for information).While I'm not exactly lobbying for this web service, we are a volunteer-driven organisation and if Tamás feels like doing it, all the merrier.

Lionel Elie Mamane committed a patch related to this issue tdf#101789 : Firebird 3 - fails to build on OSX 10.11.x with clang

Lionel Elie Mamane committed a patch related to this issue tdf#101789 Firebird 3 - fails to build on OSX 10.11.x with clang.

It has been pushed to "master": tdf#101789 work around DYLD_LIBRARY_PATH limitations on newer MacOS X.

An alternative fix has been in place ever since Stephan Bergmann commit " Allow external/firebird to be built with a custom SHELL under Mac OS X 10.11".

Bunth Tamás commited a few patches related to Firebird driver

Tamás committed a patch related to tdf#74172 issue.
Ensure prepared while setting numeric
It has been pushed to "master":
It will be available in 5.3.0. The patch should be included in the daily builds available at http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours.
More information about daily builds can be found at: http://wiki.documentfoundation.org/Testing_Daily_Builds
Affected users are encouraged to test the fix and report feedback.

Another commit is just code refactoring : get table name from resultset simply. There is a much simpler way to access table name in ResultSetMetadata in Firebird sdbc.

Bunth Tamás commited a patch for LibreOffice bug #74172 : use DECIMAL and NUMERIC data types

 Bunth Tamás commited  a patch for bug #74172 use DECIMAL and NUMERIC data types

From the list a of Firebird integration blockers there are only a few left

Listed blockers:
  bug 69949 - table editor should support the autoincrement
  bug 71009 - relationship management not implemented in GUI
  bug 71251 - Creating table from default choices via wizard fails
  bug 74019 - Any table or select query having more than 10 fields fails
  bug 74172 - no data type DECIMAL/NUMERIC 
  bug 76072 - view creation not implemented

Tensorflow and why i love it

I work for my paper for master degree (Artistic style Implementation with Tensor Flow) and i love Tensorflow

Here are a few reasons why : It's python also i can use it quite easily with my gpus (gtx) also it's distributed across servers using gRPC


Tensor Flow has a clean, modular architecture with multiple frontends and execution platforms. Details are in the white paper.

 python -m tensorflow.models.image.mnist.convolutional
I tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcublas.so locally
I tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcudnn.so locally
I tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcufft.so locally
I tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcuda.so.1 locally
I tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcurand.so locally
Extracting data/train-images-idx3-ubyte.gz
Extracting data/train-labels-idx1-ubyte.gz
Extracting data/t10k-images-idx3-ubyte.gz
Extracting data/t10k-labels-idx1-ubyte.gz
I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:900] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
I tensorflow/core/common_runtime/gpu/gpu_init.cc:102] Found device 0 with properties:
name: GeForce GTX 750
major: 5 minor: 0 memoryClockRate (GHz) 1.0845
pciBusID 0000:01:00.0
Total memory: 2.00GiB
Free memory: 1.60GiB
I tensorflow/core/common_runtime/gpu/gpu_init.cc:126] DMA: 0
I tensorflow/core/common_runtime/gpu/gpu_init.cc:136] 0:   Y
I tensorflow/core/common_runtime/gpu/gpu_device.cc:755] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 750, pci bus id: 0000:01:00.0)
Initialized!
Step 0 (epoch 0.00), 23.4 ms
Minibatch loss: 12.054, learning rate: 0.010000
Minibatch error: 90.6%
Validation error: 84.6%
Step 100 (epoch 0.12), 19.2 ms
....
Minibatch loss: 1.595, learning rate: 0.006302
Minibatch error: 0.0%
Validation error: 0.8%
Step 8500 (epoch 9.89), 20.0 ms
Minibatch loss: 1.608, learning rate: 0.006302
Minibatch error: 1.6%
Validation error: 0.8%
Test error: 0.8%

After learning Theano now i can easily convert to TensorFlow
Here are few tutorials
https://github.com/nlintz/TensorFlow-Tutorials

Firebird 3.0 is released

Firebird Project is happy to announce general availability of Firebird 3.0 — the latest major release of the Firebird relational database.

The primary goals for Firebird 3.0 were to unify the server architecture and to improve support for SMP and multiple-core hardware platforms. Parallel objectives were to improve threading of engine processes and the options for sharing page cache across thread and connection boundaries.

Alongside these aims came new strategies to improve performance, query optimization, monitoring and scalability and to address the demand for more security options. A number of popular features were introduced into the SQL language, including the long-awaited support for the Boolean data type and the associated logical predications.

Read the Press Release for more information. Please refer to the Release Notes for the full list of changes. Binary kits for Windows and Linux on both 32-bit and 64-bit platforms are immediately available for download. MacOS X and other platforms will follow.

Firebird 3 final is about to be released

The source code was just tagged in the repository, so we can expect the final release being out in the next days.

One change in the new Firebird 3 OO examples api is addition of the Services example .

Good news : Firebird 3.0 builds and works now with libreoffice

It was the comments that I forgot in download.lst (I forgot to comment 2.5.5 and enable 3.0).

I have updated the patch and now it compiles fine on ubuntu 15.10 x64
https://gist.github.com/mariuz/bcde7f783b657b2465e5

Submited new gerrit
https://gerrit.libreoffice.org/23737



Here is manual test :)
./instdir/program/soffice

Icu error with Firebird 3.0 patch , testing lode script

I started refactoring firebird 3.0 patch and there are few issues with icu linking.
There are is no more --with-system-icu from the firebird configure options (Firebird on posix is using system icu : there is no icu no more in firebird tree).

So firebird is using the icu from libreoffice folder (downloaded to build libreoffice)
I will try to compile libo with --with-system-icu

On Windows 10 i started testing the lode script for setting up libo development 

Firebird 2.5 Patch to compile with Visual Studio 2015

Final Firebird 2.5 Patch to compile with Visual Studio 2015 is now merged
https://github.com/FirebirdSQL/firebird/pull/12

I will add a gerrit for libreoffice also https://patch-diff.githubusercontent.com/raw/FirebirdSQL/firebird/pull/12.patch

Gerrit for this patch is now created and added Lionel to the reviewers

https://gerrit.libreoffice.org/#/c/23738/


update: added patch with mod_loader changes for VC14 ,
removed external/firebird/firebird-vs2013.patch.1 is not needed anymore in 2.5.5 (applied upstream) ,

modified UnpackedTarball_firebird.mk apply firebird-msvc14.patch.1 only on Windows

ps: in Firebird tree i will send a pull request with icu small changes needed also new solutions for VS 2015 were merged earlier

Firebird 3.0 and MSVC14 build (good news)

I have tested the Firebird 3.0 and MSVC14 compiler and builds ok

https://github.com/FirebirdSQL/firebird/tree/B3_0_Release/builds/win32/msvc14


Tomorrow I will test also with firebird 2.5.x branch and i will check what patches are needed

Instructions for building it are located here

https://github.com/FirebirdSQL/firebird/blob/master/doc/README.build.msvc.html

and on the website

http://www.firebirdsql.org/en/building-the-code/

Let's go back to the c , Firebird embedded backup service works

I have used an c based example for services api and seems that it works without a need for auth user



gcc backup_service.c -lfbembed

./a.out employee.fdb employee.fbk

Attach succeed

Start succeed

gbak:readied database employee.fdb for backup gbak:creating file employee.fbk

...

gbak:writing SQL roles gbak:writing names mapping gbak:closing file, committing, and finishing. 74752 bytes written

I will split the backup service function in two one for backup and one for restore then i will compare with my service example , maybe is a buffer truncation issue

Update firebird to version 2.5.4 is applied into LibreOffice master

There was a small issue with tinderbox and was reverted and then re-applied to master.

Rebuilding of indexes function is now removed from master (Using firebird backup format for .odb there is no need for rebuild indexes).

JDK issues on OSX , ran the tests without Java

To run all the dbaccess tests On OSX I had to install jre 6 then jdk/jre 8 but still
dbaccess tests wouldn't run java popup still appeared even if the java -version showed
the proper 8 version , so for the moment i have disabled the java tests

./autogen.sh --without-java

Update: Tests are finished and Firebird 2.5.4 update patch on gerrit is up for review
https://gerrit.libreoffice.org/16703

Tested on OSX 10.10.4 , Win 10 , Ubuntu 14.04

Debugging with ddd : crashes in rtl_ImplGetFastUTF8UnicodeLen

Found it where it crashes at line 645 in Connection.cxx

OUString aData(pResults + 3,
                           isc_vax_integer(pResults, 2),
                           RTL_TEXTENCODING_UTF8);
            

and here is the backtrace at line 634 in ustring.cxx

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff7b7aebd in rtl_ImplGetFastUTF8UnicodeLen (pStr=0x7ffffffff000 , nLen=11582, ascii=0x7fffffffc4b8) at /home/mariuz/work/libo/sal/rtl/ustring.cxx:634
(gdb) frame 4
#4  0x00007fff96c9bce2 in connectivity::firebird::Connection::runBackupService (this=0x7fffac0a1400, nAction=1) at /home/mariuz/work/libo/connectivity/source/drivers/firebird/Connection.cxx:645
(gdb) frame 24
#24 0x00007ffff195e906 in SalFrame::CallCallback (this=0x10a7680, nEvent=22, pEvent=0x15fd5d0) at /home/mariuz/work/libo/vcl/inc/salframe.hxx:246
(gdb) frame 23
#23 0x00007ffff1586e00 in ImplWindowFrameProc (_pWindow=0x10a6ce0, nEvent=22, pEvent=0x15fd5d0) at /home/mariuz/work/libo/vcl/source/window/winproc.cxx:2578
(gdb) frame 5
#5  0x00007fff96c9c99c in connectivity::firebird::Connection::documentEventOccured (this=0x7fffac0a1400, _Event=...) at /home/mariuz/work/libo/connectivity/source/drivers/firebird/Connection.cxx:814
(gdb) frame 4
#4  0x00007fff96c9bce2 in connectivity::firebird::Connection::runBackupService (this=0x7fffac0a1400, nAction=1) at /home/mariuz/work/libo/connectivity/source/drivers/firebird/Connection.cxx:645

Backup file firebird.fbk file inside odb is 0 length

That could be the reason gerrit wasn't merged Connection::runBackupService didn't quite worked:

(Currently doesn't actually work, the backup service fails with gbak:Exiting before completion due to errors and no further error messages).

I need to debug the isc_action_svc_backup error

On odb save event now saves fbk instead of fdb

XDocumentEventListener documentEventOccured was registered for when save button is pressed

It was done on this line in Connection.cxx :
xBroadcaster->addDocumentEventListener(this);

The issue was on this part of the code 

xInputStream = OStorageHelper::GetInputStreamFromURL(m_sFirebirdURL, xContext);

Now replaced with

xInputStream = OStorageHelper::GetInputStreamFromURL(m_sFBKPath, xContext);

m_sFirebirdURL is the original fdb path 

m_sFirebirdURL = m_pDatabaseFileDir->GetFileName() + "/firebird.fdb";
m_sFBKPath = m_pDatabaseFileDir->GetFileName() + "/firebird.fbk";